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Abstract 

Fatal loss-of-control accidents have been directly related to in-flight airframe icing. The prototype 
system presented in this report directly addresses the need for real-time onboard envelope protection in 
icing conditions. The combination of prior information and real-time aerodynamic parameter estimations 
are shown to provide sufficient information for determining safe limits of the flight envelope during in- 
flight icing encounters. The Icing Contamination Envelope Protection (ICEPro) system was designed and 
implemented to identify degradations in airplane performance and flying qualities resulting from ice 
contamination and provide safe flight-envelope cues to the pilot. The utility of the ICEPro system for 
mitigating a potentially hazardous icing condition was evaluated by 29 pilots using the NASA Ice 
Contamination Effects Elight Training Device. Results showed that real time assessment cues were 
effective in reducing the number of potentially hazardous upset events and in lessening exposure to loss 
of control following an incipient upset condition. Pilot workload with the added ICEPro displays was not 
measurably affected, but pilot opinion surveys showed that real time cueing greatly improved their 
awareness of a hazardous aircraft state. The performance of ICEPro system was further evaluated by 
various levels of sensor noise and atmospheric turbulence. 

1.0 Introduction 

Ice formations on aircraft can negatively affect performance, stability and control characteristics, and 
handling qualities. Accidents that have occurred due to these issues have been documented by the 
National Transportation Safety Board (NTSB) and the Eederal Aviation Administration (EAA) over many 
years. In a recent study (Ref. I) of aircraft icing accidents and incidents reported between 1978 and 2002, 
loss of control events were usually preceded by stall and more likely to occur during approach and 
landing. This is the busiest phase of flight where historically over 50 percent of all accidents occur, and 
where human error is responsible for approximately 75 percent of those accidents (Ref. 2). Ice 
contamination, on a critical airframe surface such as the wings and tail, which might be due to an Ice 
Protection System (IPS) failure, encountering an icing certification exceedance condition, or operator 
error can have a detrimental effect on aircraft performance, stability and control, and handling qualities. 
This situation is further exacerbated by the required changes in airspeed and configuration as the pilot 
slows and configures during the descent, approach, and landing phases. 

One notable accident referenced above involved an ATR 72 that flew through SLD icing conditions 
near Roselawn, Indiana, on October 31, 1994. Even with the IPS operating normally, the aircraft 
experienced a roll control anomaly that led to the loss of control and impact into the ground. This accident 
and others since then have led to numerous safety recommendations from the National Transportation 
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Safety Board (NTSB) (Refs. 3 and 4) and the Commercial Aviation Safety Team (CAST) (Ref. 5) and 
motivated research activities within NASA and the FAA. 

In its Tailplane Icing Program (TIP), NASA employed various artificial ice shapes, which were 
attached to the horizontal tail plane (Ref. 6) of a research aircraft. Numerous test maneuvers were 
conducted where wing flap settings, engine power, and flight speed were varied. The results showed a 
considerable decrease in longitudinal stability and elevator control effectiveness as a function of wing flap 
angle, power setting, and airspeed. In extreme cases, a tail stall upset would occur, which resulted in an 
uncontrollable nose down pitching motion. Airframe ice contamination also increases drag, reducing 
climb performance. In another research program, NASA sponsored an activity called the Smart Icing 
Systems led by the University of Illinois, Urbana-Champaign (UIUC). The Smart Icing Systems took a 
systems level approach to detecting icing conditions, operating ice protection systems and alerting the 
flight crew to the status of the aircraft (Ref. 7). 

Most recently NASA issued Research Announcement (NRA) NNH06ZEA001N, Appendix B, of the 
Aviation Safety Program. The research announcement was defined under the Integrated Vehicle Health 
Management (IVHM) Project, topic IVHM 3.1, Environmental Hazards, which are caused by the “Effects 
of Icing on Aircraft State”. The function of IVHM is to manage the health of the vehicle by detecting, 
diagnosing, prognosing, and mitigating in-flight environmental hazards. The main objective was to 
develop tools that measure the state of the aircraft in atmospheric icing conditions and assess the effects 
of icing on the aircraft for various environmental icing conditions. The original scope for this work 
included simulation and experimental testing but was later constrained to pilot in the loop simulations. 

The goals for this work were: 

1. Develop a system that performs real time assessment of icing effects on aircraft stability, control, and 
performance. 

2. Provide the flight deck with flight guidance cues, configuration and envelope limits, and related state 
assessment messaging to avoid upset conditions and loss of control. 

3. Enable the pilot to make better tactical decisions regarding flight speed and configuration selections 
to maintain the aircraft in a safe envelope of flight. 

The University of Tennessee Space Institute (UTSI) in partnership with Bihrle Applied Research 
(BAR) completed a cooperative research effort addressing the IVHM function, objective, scope, and 
goals. This effort culminated in the development of a real time vehicle state assessment system, which 
was described by Gingras et al. (Ref. 8) and is referred to as the Icing Contamination Envelope Protection 
system (ICEPro). The ICEPro system was developed with the philosophy that a knowledge-based system, 
which compares expected aircraft stability and control characteristics with those existing in real time, 
could effectively provide envelope protection cues and alerting messages to help the pilot keep the 
aircraft flying safely. The knowledge base consists of a model for the baseline or clean aircraft, and an ice 
contamination model with known safe flight envelope limits as demonstrated by either wind tunnel or 
flight test. Angle of attack (AOA) cues, which are driven by ICEPro, show the pilot safe limits that will 
protect against either a wing stall or tail stall upset event. Messages alert the pilot to losses in stability and 
control as a function of flight configuration and condition, and to configuration limits as a function of 
icing and wing flap setting. 

The algorithms in ICEPro were initially developed and tested in a desktop simulation environment, 
and then integrated and further developed in the NASA Ice Contamination Effects Elight Training Device 
(ICEETD). The flight models used in simulation to develop ICEPro for testing and evaluation were based 
upon the characteristics of the NASA Twin Otter icing research aircraft in a no-ice baseline condition, 
and a failure of the ice protection system (IPS). The failure case was defined by the ice accretions that 
would form during a 22.5 min PAR 25 Appendix C icing encounter as calculated by LEWICE version 
2.0. The data for the simulator came from a previous NASA project where a 6.5 percent scale Twin Otter 
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model was tested in the Wichita State University 7x10 Low Speed wind tunnel and BAR’s Large 
Amplitude Multi-Purpose (LAMP) rotary balance wind tunnel using scaled ice shapes (Ref. 9). Flight 
data from aircraft testing with baseline and failure case ice shapes was also gathered to refine the 
simulation model data to improve the match full scale flight test results (Ref. 10). 

After development and testing in a desktop environment, ICEPro was integrated with the NASA 
ICEFTD. Researchers from BAR and UTSI completed final development and integration testing to ensure 
readiness for the planned pilot evaluations. A final check of test readiness was then performed by having 
two experienced NASA test pilots complete the evaluation profile, and all required pre-test preparation 
and training. This included a review of: the test plan, NASA web based icing training, standard operating 
procedures (SOP) for the Twin Otter, the NASA Task Load Index or TLX workload method (Ref. 1 1), a 
tutorial on flying with ICEPro displays (only for the pilot using the experimental displays), and post-test 
survey questionnaire. One NASA pilot was selected to represent the control group and flew with baseline 
displays. The other was selected to represent the experimental group and flew with the ICEPro modified 
displays. Before the test began, each pilot was briefed on the entire test process. The test protocol at the 
time called for minimal training in aircraft handling with hazardous ice formations, somewhat similar to 
what a typical pilot would receive in an aircraft transition course. However, once testing began, it became 
apparent that the flight task was so difficult that without practice, there was a high probability of a 
random “crash” regardless of the displays used. It was apparent that a different approach would have to be 
taken. After assessing the pilot’s comments from the debriefings, it was therefore decided to modify the 
test protocol and train the pilots in both groups to proficiency for executing the challenging approach task 
using their respective flight displays. Utility benefits could then be assessed by comparing pilot 
performance, workload, and opinions between the control and experimental groups who were flying the 
same task with the same training and preparation, but with different flight displays. This adjustment in the 
test plan was an essential change, and would not have been discovered without this pre-test. The NASA 
pilots also completed the NASA TLX forms and answered all the survey questions during their respective 
evaluations. A few small changes in the survey questions were also required to eliminate ambiguity 
problems. After these adjustments were made to the test plan, pilot-in-the- loop (PIL) evaluations of 
ICEPro were scheduled at the Embry Riddle Aeronautical University (ERAU), Daytona Beach, Florida. 

However, the simulation could not duplicate all of the real-world conditions that could have first- 
order effects on system performance, such as those due to real atmospheric turbulence. Therefore, in order 
to assess system performance under these conditions and minimize the technical risk of an eventual flight- 
test validation program, the effects of atmospheric turbulence on ICEPro were studied. This led to a one 
year extension awarded under NASA’s Vehicle Systems Safety Technologies (VSST) project. 

This contractor report summarizes the results of this research, which focused on the concept, design, 
development, implementation, and evaluation of the real-time vehicle state assessment system. This 
evaluation included pilot cueing for flight envelope protection, and evaluation of the effectiveness of the 
ICEPro system in piloted simulations using the ICEFTD. Sections 2.0 to 9.0 focus on the concept, design, 
development, and implementation of ICEPro. Section 10.0 explains the methodology used for pilot-in- 
the -loop testing and overall experiment design. Section 1 1.0 presents the results and discussions of pilot- 
in-the-loop testing. Section 1 2.0 discusses the atmospheric turbulence test methodology and finally 
Section 13.0 cover conclusions and recommendations. There are five appendices attached to this report. 
Appendix A is the Symbols List. Appendix B includes the post-test survey questionnaire used in the 
pilot-in-the -loop simulations. Appendix C includes the results of the post-test survey questionnaire. 
Appendix D includes the progress report of the effects of atmospheric turbulence on ICEPro through the 
funded one year extension. Appendix E includes the atmospheric turbulence test plan. 
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2.0 ICEPro Concept 

The ICEPro concept is best summarized by the hypothesis posed by the development team during 
preliminary stages of the design process. 


By the real-time processing of measured state and control information, estimates of stability 
and control can be used in conjunction with a knowledge base, comprised of a-priori data, to 
provide pilots with envelope-limiting cueing in order to avoid loss-of-control or adverse 
conditions resulting from in-flight icing. 

The ICEPro system was designed and implemented to identify degradations in airplane performance 
and flying qualities resulting from ice contamination and to provide safe flight-envelope cues to the pilot. 
The ICEPro concept relies on the actions of a proficient pilot responding to appropriate cueing and 
information on the primary and secondary displays in the cockpit. It is believed that, with these cues, the 
pilot can make informed decisions as to how the airplane should be flown. The ICEPro concept provides 
envelope protection by issuing advisories, cautions, and warnings in the form of airspeed limits, angle of 
attack limits, flap position limits, and control effectiveness status. The indications are intended to be 
consistent with typical advisories and warnings that appear on flight displays, so that they are intuitive 
and effective. Eigure 1 provides a graphic illustrating the cueing that is used in the ICEPro system. 

When the system does not detect significant change from the nominal prior airplane characteristics, 
no cuing or messages appear and the instruments display the nominal flight information. When the system 
senses icing contamination that exceeds the system thresholds and triggers the latching logic, cues and 
possibly messages appear in conjunction with the normal display information. 


Color-coded 
Velocity Limit 
Carets 


Color-coded 
AOA Limit 
Brackets 


Color-coded 
Flap Position and 
Flap Limit Indicator 


Color-coded 
Control Surface 
Condition Indicators 


Control Surface 
Position Indicators 


Color-coded 
Stability & Control 
Caution & Warning 
Lights 




Color-coded 
Flap Position 
Indicator 



Climb Limit 
Due to Icing Indicator 


Color-coded 
Stability & Control 
Caution & Warning Messages 


Figure 1. — ICEPro Envelope Protection Cues on the Primary Flight Display (left) and Flight Control Status 
Page (right) 


NASA/CR— 2014-218320 


4 




There are three types of information that are provided when icing contamination is determined. See 
Figure 1 for illustration. 

• Airspeed limit carets and angle of attack limit brackets. 

The airspeed limit carats for icing are to the right of the speed tape and annotated with a * to 
indicate icing. The angle of attack limit brackets are superimposed on the attitude indicator. The 
upper bracket indicates the wing stall AOA. The lower bracket indicates the tail stall AOA. The 
color of the brackets change from white to amber when the AOA is less than 2° lower than stall 
AOA and then change to red when the AOA reaches the stall AOA. The brackets will also flash 
for several seconds with each color transition, white to amber, and amber to red. 

• Status messages 

These will appear, both on the primary flight display and on the Flight Control Display (FCD) 
page in the message box. They indicate that there is a control and/or stability degradation in the 
indicated axis. There is also a message that appears whenever the single -engine climb capability 
is outside of safe limits. A flap limit message will also appear when the flaps should be retracted 
to a safer deflection. 

• Flight control status indicators 

The FCD page includes a rough diagram of the airplane showing its control surfaces. The 
surfaces are colored and the color will change whenever degradation is determined in their axis. 
With no degradation, the surfaces are green and turn amber when there is a caution displayed for 
that axis and red when there is a warning. The color-coded displays are driven by current 
estimates of control effectiveness and thresholds defined inside ICEPro. 

The basic design architecture of the system is shown in Figure 2. The ICEPro System has three modes 
of operation: a MONITOR mode that provides initial detection of in-flight icing, IDENTIEICATION or 
ID mode that provides a diagnosis of the condition, and a REPORTING mode, providing the prognosis of 
the condition and mitigation through cueing. Details of the system design can be found in Reference 8. 

Eigure 3 presents a simplified flow chart. Details, such as latching/delatching logic and specific logic 
for driving each of the caution/warning messages have been omitted to reduce the complexity of the 
diagram. As indicated in the diagram, the system is heavily dependent on input from two components: 

1) The Dynamic Inversion Control Evaluation System (D-ICES), an algorithm that computes what the 
control surface deflections should be for a clean aircraft compared to what the control deflections actually 
are, and 2) the Real-Time Parameter IDentification (RT-PID) routine. Results from these components are 
used to determine an Icing Severity Parameter (ISP) which is used in logic for issuing pilot cues and 
warnings during the reporting mode. 



Figure 2. — ICEPro System Architecture 
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Figure 3. — Simplified ICEPro System Flow Chart 
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3.0 Flight Characteristic Predictions 

Flight predictions are made inside ICEPro using a flight model of a DeHavilland DHC-6 Twin Otter. 
This flight model was derived from a comprehensive nonlinear 6-degree-of-freedom simulation that was 
developed by Bihrle Applied Research and engineers at the NASA Glenn Research Center (GRC) 

(Ref. 9). The model was based on data from extensive wind-tunnel tests using subscale models of the 
Twin-Otter and supplemented with flight data provided by NASA. This model covers the entire flight 
envelope including stall and post-stall flight regimes with and without icing effects resulting from ice 
accretion on the main wing, horizontal tail, and vertical tail. During previous efforts, the model had 
undergone extensive validation against flight data to achieve high confidence predictions of aerodynamic 
characteristics in clean configuration, as well as the aerodynamic characteristics from an iced 
configuration representing an IPS failure situation. Figure 4 contains a flow chart of the process used 
during this effort to validate the aerodynamic simulation modeling. 

Figure 5 contains an example of the resulting comparisons of model output for normal force 
coefficient, axial force coefficient, and pitching moment coefficient, with flight data from an elevator 
doublet maneuver on the clean airplane. This process was used to update the aerodynamic simulation 
model based on flight data. This effort resulted in the knowledge base system or database referred to in 
this work. 


A, p. Speed, Altitude 
Angular Rates (P, Q, R) & 



Figure 4. — Process used for Aerodynamic Model Validation 
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Figure 5. — Example of Comparisons of the Simulation Model and 
Flight Aerodynamic Coefficients 

4.0 Dynamics Inversion Control Evaluation System 

This technique, the Dynamics Inversion Control Evaluation System (D-ICES) makes use of the 
system knowledge base to predict the control deflections necessary to achieve the vehicle motion. D- 
ICES tracks the current pitch, roll, and yaw rate in each axis using a proportional, integral, and derivative 
(PID) control with lead compensation, Eigure 6. With this tracking, the amount of angular acceleration 
required to match the aircraft motion is determined. Using information from the knowledge base, 
predictions of the current “clean” stability of the vehicle are entered into the aircraft equations of motion 
along with weight and balance information and the tracking accelerations to compute a “residual” 
acceleration vector. 

By assuming this residual acceleration is a direct result of control movement, a prediction of control 
deflections can be obtained (Eq. (1)). Changes in the vehicle aerodynamic state from the “clean” can be 
detected by tracking these predictions of the control deflection in flight and comparing them to actual 
control deflections. 


■^Measured ^-^Predicted ^^Predicted 

^Predicted ~ ^Control -^Measured ~ ^Stability '^'Predicted 


( 1 ) 
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Figure 6. — Tracking algorithm for acceleration signal input to the dynamic inversion scheme 


The actual comparisons in the logic are based on the calculated Theil inequality coefficients (Ref. 12) 
for velocity and elevator deflections. The Theil inequality coefficients are used in a moving time window 
to evaluate how well the extracted data is matching the expected levels during that time period. These 
statistics form the basis for the Icing Severity Parameter to be discussed in Section 1 1.0. The form of the 
Theil inequality coefficients is shown in Equation (2). 










LSE 
y RMS 


( 2 ) 


Preliminary indications of icing can be assumed when the Theil inequality coefficients for these 
parameters exceed a threshold level that has been calibrated through simulated exercise of the system. 


5.0 Real-Time Model Estimates 

Because the system compares model predictions described above with the current state, real-time 
estimates are used. These estimates are computed by invoking real-time parameter identification (RT- 
PID). The RT-PID algorithm being employed was developed by Morelli (Ref. 13), and is based on a 
frequency domain approach implemented in version 2.0 of System Identification Programs for Aircraft 
(SIDPAC).(Refs. 14 and 15) The approach uses a linearized model of the vehicle dynamics. Equation (3), 
where A, B, C, and D matrices containing stability and control derivatives. 

jc(r) = Ax(f) + BM(r) jc(0) = x,, 

( 2 . 

y (f) = Cx(^t^ + Du (f) 

Time-varying estimates of the stability and control derivatives are determined using measured airplane 
control surface deflections and measured states from air data, angular rates, and attitude. Outputs are 
rotational and translational accelerations. To determine the estimates, a cost function is formulated in the 
frequency domain from the Eourier transform of the model in Equation (3), 
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The cost function is 


jco^(co) = A^(co) + Bu{p) 
j ( co) = ( co) + Dm ( co) 


( 4 ) 


h 


2 


^ |jo)„ Xj^ for m frequencies 

n-\ 


( 5 ) 


where and bk are the ^h row of matrices A and B, respectively, and (n) = (co„) , (n) = (co„ ) . 

Each line in Equation (4) can he analyzed separately in this way, which implements an equation-error 
formulation as desciihed in Reference 14. The least squares cost function in each case can he formulated as 


J =^{Y-XQf [Y-XQ] 
where ^ indicates complex conjugate transpose. 


mh (1) 


(l) 


X = 


JVk 


x^ (m) (m) 


( 6 ) 


( 7 ) 


and the unknown parameters from A and B are contained in the parameter vector 0. The least squares 
parameter vector estimate is obtained as the value of 0 that minimizes the cost function in Equation (6) 
(Ref. 13). 


0 = 


Re 


^Re(x'''T) 


a = — 
m 


cov(0) = D (0-0)(§-0) 

Ay-xef(y-M) 


Re 




( 8 ) 


Performing the identification in the frequency domain is advantageous for this application because of 
computational efficiency, robustness to noise and data dropouts, and reliable confidence bounds from the 
covariance matrix. To ensure reliable estimates, the ICEPro features a self-checking algorithm that 
examines the confidence bound of the parameter estimates. Table 1 lists the parameters that are being 
identified in real-time. 


TABLE 1.— AERODYNAMIC PARAMETERS IDENTIEIED BY RT-PID 
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If confidence of the estimated parameters falls below acceptable levels due to lack of data information 
content (low pilot activity), an automated excitation of the aircraft control surfaces is triggered to improve 
data information content. This interface with the subject vehicle is performed as though an autopilot is 
driving control deflections using multi-frequency orthogonal inputs (Ref. 15). Amplitudes of the control 
deflection inputs are adjusted to provide sufficient excitation in each axis while minimizing vehicle 
motion. 

6.0 System Implementation 

The ICEPro system was developed and deployed in Bihrle Applied Research’s D-Six PC-Based 
simulation environment. All of the ICEPro functions, including the software for driving the flight displays 
are provided in a single computer that runs the software in real time. The system can be interfaced with a 
desktop simulation, NASA’s Ice Contamination Effects Flight Training Device (Ref. 16), ICEFTD 
(Figure 7), or mounted in the NASA Twin Otter aircraft for flight demonstrations. The system includes all 
of the Data Acquisition and Executive functions which control the various modules that make up the 
system. 

The data acquisition portion receives data from the data bus on the airplane or from an emulation of 
the airplane data bus if it is working with the simulator. It then makes the appropriate data available to the 
various other modules for processing. The Executive portion provides all of the logic and calculation 
modules required to determine the icing state and what cues should be provided to the pilot. 

The ICEPro system was first developed using a desktop simulation running in D-Six. The module 
development and all of the initial check out and system calibration were executed in this manner. Once 
the system was sufficiently mature to permit piloted evaluations, it was migrated to the ICEFTD. On the 
desktop implementation, the ICEPro modules and the Twin Otter simulation were run on the same 
computer. For the ICEFTD implementation, a separate computer was used for the ICEPro modules. 
Communication with the ICEFTD was accomplished over an Ethernet connection. This implementation 
was used to facilitate an eventual implementation in the actual airplane. If this were to occur, the same 
computer would be similarly connected to the airplane’s data acquisition system. 



Figure 7. — NASA's Ice Contamination Effects Fiight Training Device (ICEFTD) 
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The ICEFTD piloted simulations provided synthetic environment to assess and tune the final 
thresholds and latching logic for the cues and messages. They also afforded the opportunity to assess the 
effect of control surface excitation on the pilot. The piloted simulation was also instrumental in evaluating 
the appearance and validity of the caution/warning messages, and the development and evaluation of 
aural and tactile cues. These messages and cues were incorporated for the guest pilot evaluations that 
were subsequently performed following the development stage. 

7.0 Engineering Station 

An engineering station was set up that enabled the operator to set key parameters of the ICEPro 
system, monitor the performance of the system, and interact with the implemented logic during the 
development and evaluation phases of the program. The engineering station consisted of a set of 
Graphical User Interfaces (GUIs) that were displayed on two monitors attached to the computer running 
ICEPro. The first window (Figure 8) shows the status of the system. It shows in which mode the system is 
currently operating (Monitor, ID, or Report), whether the controls are being excited, the duration and 
magnitude of the excitations in each axis, as well as the time since reset of both the D-ICES and Real 
Time Parameter Identification (RT-PID) modules. There is also a window that displays system status 
messages and buttons to reset the system modules and settings. The duration and magnitude of the surface 
excitations can be adjusted in this GUI and the excitation can be manually triggered when the system is in 
the ID or Report modes. It is also possible to manually reset either the D-ICES or RT-PID routines from 
this window. 

There are a series of default thresholds, time delays, etc. that are used by the ICEPro logic modules 
that are preloaded with the system. All of the default values can be overridden by inputting new values in 
an included text file. There are two buttons at the bottom of the GUI to either update these system settings 
from the text file, or to reset them to the default values. 



Figure 8. — System Status Window 
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The D-ICES status window, Figure 9, shows the a priori model values of the longitudinal and lateral- 
directional derivatives for both the clean and iced (Failure IPS icing shapes) airplanes at the current 
aircraft states. These linearized derivatives are extracted at the current state from the stored nonlinear 
database contained in the D-ICFS routines. There is also a similar window for the RT-PID routines, 

Figure 10 that shows comparable real time PID derivatives. The values represented in Figure 9 and Figure 
10 are only meant to illustrate the GUI and are not representative. 

The RT-PID module only runs in the ID and REPORT modes, therefore, the values are not updated 
when the system is operating in MONITOR mode. The operator can compare the values from the D-ICES 
and RT-PID status windows to evaluate the differences between the a priori clean or iced values, and the 
actual values as determined by RT-PID. A comparison was plotted off-line and is shown in Figure 11. 



Figure 9. — D-ICES Status Window 
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Figure 10. — RT-PiD Status Window 
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Figure 11. — Comparison of RT-PID Resuits (red) with D-ICES Prediotions (biue), with and without Contamination 


Because the ICEPro system modules are running within a D-Six project, all of the normal D-Six 
interface tools are also available at the engineering station. As a result, any of the aircraft variables that 
are passed from the ICEFTD are available to monitor in real time listings, strip charts, or as post-run 
engineering plots. Pertinent ICEPro variables are similarly available for real time monitoring and post-run 
analysis. At the end of a run, the time histories of these variables can be recorded for subsequent analysis 
or for play back. The system also provides strip charts of any of the RT-PID derivative values along with 
their expected values and the calculated error bands. 

8.0 System Calibration 

A considerable amount of effort was expended fine-tuning the system to provide adequate warning 
and guidance in an icing encounter without any false indications or nuisance messaging. These efforts 
included three main areas: determining which terms to include in the ISP calculation, tuning thresholds 
and latching and delatching logic, and developing reset logic for the RT-PID routines. 

8.1 Icing Severity Parameter Determination 

As shown in Equation (9), the ISP was formulated as the sum of the difference between the expected 
(D-ICES clean aircraft) and actual (RT-PID derived values) derivative values divided by the difference 
between the a priori clean and iced (Failure IPS icing shapes) values for the selected derivatives. The ISP 
is a single number with a range between 0 and 1 . Experimental testing of the system determined that the 
threshold for ice contamination detection was an ISP of 0.5. During developmental testing, it was found 
to be impractical to use all of the RT-PID derivative terms. For example, the yawing moment terms for 
this aircraft did not change appreciably with icing condition, and did not prove useful to include in the 
ISP calculation. The damping terms, such as C,„^ and Cj^, which did show differences with icing, were the 
most difficult for RT-PID to obtain with great accuracy. These damping terms proved to adversely affect 
the robustness of the ISP calculation and therefore were not used. 

n 

I 

1=1 

ISP^ — 

n 

The control terms, i.e., the 5^, and 8^ moment derivatives, were generally determined with great 
accuracy, and showed appreciable changes with icing condition. Consequently, these terms were used in 
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the ISP calculation. The tenns for longitudinal and lateral stability, C„,„ and C;p were also used. The final 
terms included in the ISP calculation were: 


C 

CNa 


C/3 

C/8a 


Additional logic was also added to consider the relative predicted error in each term, in the form of 


- P/D _ error 
RT - PID _ value 


( 10 ) 


Where, the term {RT-PID_en:ox) is the 2 sigma confidence hound of the estimate, and the term {RT- 
P/D_value) is the value of the estimate. If the relative predicted error for any term became larger than an 
experimentally determined threshold, the term was dropped from the ISP calculation. As a result, any 
term whose error represented a sufficiently large portion of its total value would not adversely influence 
the ISP calculation. As will be discussed further, the relative error in the pitching moment due to elevator 
deflection derivative was such a critical value to the ISP calculation (one of the earliest indicators of 
degraded RT-PID results), that its relative error was used as one of the reset criteria to the RT-PID 
routines. 


8.2 Thresholds and Latching 

For each change in mode of operation and messaging, there were thresholds that needed to be met 
before the change could occur or the message appeared. For example, an experimentally derived ISP 
threshold of 0.5 is required for the system to transition from monitor mode to ID mode. The value of 0.5 
was selected so that the ISP threshold could be satisfied with only one term in the ISP formulation. There 
was also latching and delatching logic that specified the condition had to be met for a specified period of 
time before it would appear and, conversely, the condition had to disappear for a period of time before it 
could be removed. This logic was incorporated so that the messages would not “flash” off and on if the 
conditions were right on the edge of display thresholds. 

8.3 RT-PID Reset 

As was mentioned earlier, it was found necessary to force a reset of the RT-PID routines periodically 
for best operation. The RT-PID calculations make use of past values which works well if there is no 
change in the aircraft configuration or flight condition. However, when there are changes, the time 
histories of past values can incorrectly influence the calculations at the new configuration and/or flight 
condition. Consequently, a reset, which flushes the buffers and essentially restarts the calculations, was 
required whenever those conditions occurred. In addition, evaluations of the system showed that better 
results would occur with periodic resets even when no significant changes occurred. For the evaluation 
phase, the system was set to do an RT-PID reset after 50 sec if one had not occurred for either of the other 
two specified reasons. While control excitations were not directly commanded upon reset, they generally 
did occur shortly after the reset unless there was sufficient control activity by the pilot. 

8.4 Conditions for Warning/Caution Messages 

Messages that warned of deficiencies in a particular axis or of aircraft performance limitations were 
displayed on the PFD. The conditions under which these messages would appear, and whether they 
appeared as cautions (amber messages) or warnings (red messages) were based on the amount of 
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degradation from the baseline clean airplane values. Whenever a caution or warning message is displayed, 
the corresponding primary control surface on the airplane representation on the FCD page is also colored 
appropriate to the message (amber for cautions and red for warnings). The caution or warning message is 
also repeated in the message section of the FCD page. 

The initial trials of the ICEPro system considered stability and damping terms in each axis, as well as 
control effectiveness terms to determine when a caution or warning message should be displayed. In 
practice, it was found that meaningful stability and damping thresholds were rarely attained. Therefore, 
these terms were dropped and only the control effectiveness terms were used. Consequently, the 
conditions that will generate a pitch, roll, or yaw degraded message are as shown below (because the 
control derivatives are all negative terms, the inequality signs may seem reversed): 

0.50 • Cm^_no-ice ^ C„i^ < 0.25 • Cm^ _no-ice , then amber PTCH DGRD and amber 5e on FCD 

Cniie ^ 0.25 • Cmse_no-ice , then red PTCH DGRD and red de on FCD 

0.50 • CnSr _ _no-ice ^ Cnsr < 0.25 • Cnsr _no-ice , then amber YAW DGRD and amber 8r on FCD 

Cnsr ^ 0.25 • Cn&r_no-ice , then red YAW DGRD and red 5r on FCD 

0.50 • Ci^_no-ice ^ Ci^ < 0.25 • C/ 5 „ _no-ice , then amber ROLL DGRD and amber 5a on LCD 

Ci^ > 0.25 • C/g„ _no-ice , then red ROLL DGRD and red 8a on LCD 

Calculations of single engine rate of climb capability were made using the simulation data and Twin 
Otter Airplane Llight Manual values. Tables of these values as functions of flap deflection, ice condition, 
and altitude were stored in the ICEPro routines. Whenever the system was in the Reporting mode and the 
flight and icing condition resulted in predicted single engine rate of climb levels below specified 
thresholds, a climb limit caution or warning message would appear on the PLD and in the PCD message 
box. In addition, a corresponding flap limit message would appear and the flaps representation on the 
PCD page would be colored either amber or red, corresponding to caution or warning messages. This was 
meant to alert the pilot that the flap deflection should be reduced for safe operation in the event of an 
engine failure. The thresholds used during the evaluations were: 

0 ^m < SingEngROC < 100 ^m, then amber CLMB LIM and FLAP LIM 

SingEngROC < 0 ^m, then red CLMB LIM and FLAP LIM 

8.5 Stall Warning and Decluttering 

A stall warning was added that consisted of a large red STALL message appearing near the middle of 
the PFD and an aural two-note tone. The aural warning would come on whenever the calculated stall 
angle of attack was attained, and the message would appear on the PFD when the angle of attack was 2° 
above the calculated stall angle. During the system development, it was found that in the low-speed 
highly dynamic environment present during stall, RT-PID was unable to keep up with the rapid change in 
aerodynamic characteristics. Extraneous waming/caution messages were observed but would go away 
after the airplane was stabilized. Rather than spend significant time refining the thresholds and logic, it 
was decided, based on pilot evaluations during the development, that it was best to eliminate the 
messaging entirely during a stall and recovery. This allowed the pilot to concentrate on recovering the 
airplane without the distractions. Consequently, logic was incorporated to “declutter” the PLD by 
eliminating all ICEPro related messaging during this period. The angle of attack brackets and speed carets 
did remain on the PLD, since these would aid the pilot during the recovery effort. While these cues do 
depend on the RT-PID results, the ISP calculation is filtered to yield a more or less steady-state level. 
Therefore, the ISP calculation is marginally influenced by the rapid changes during a stall and recovery. 
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8.6 Accommodations for Pilot Evaluations 

In preparation for the piloted evaluations that took place during August 2009, several additions were 
added to the simulation. To represent to the pilot an icing protection system failure, a message stating 
“IPS failure” was displayed in the FCD message hox along with an aural cue. A stick shaker and stick 
pusher function were added to the longitudinal yoke loader. These were configured to operate in a manner 
consistent with real-world shakers and pushers and would act as a forceful cue to the pilot that wing stall 
was about to, or had occurred. The shaker was sufficiently rapid and forceful that it could not he missed 
during the most intense flight condition. The shaker was configured to come on at the same time as the 
angle of attack brackets turned from white to amber, which was 2° below the stall angle of attack. The 
pusher was forceful enough that it would force the airplane back to an unstalled angle of attack. 
Monitoring when the shaker or pusher was activated proved to be an important tool in interpreting and 
assessing the performance during post-flight analysis. The stick pusher came on at approximately the 
same time that the stall message appeared on the PFD. 

9.0 Pilot in the Loop Test Methodology 

9.1 Experiment Design 

The purpose of the pilot in the loop evaluations was to determine if ICEPro had utility for mitigating 
a potentially hazardous icing encounter (Ref. 17). The independent parameters were the two displays 
(baseline, and ICEPro with envelope protection), and the dependent parameters were technical 
performance, workload, and pilot opinion. An ICEPro display is shown in Figure 1 . A baseline display is 
Figure 1 without airspeed limit carets, angle of attack limit brackets, stall messages, and flight control 
status indicators. A nonparametric, independent measures experiment was therefore selected where 
differences in the dependent variables were analyzed using the Mann-Whitney “U” (MWU) test (Ref. 18). 
A critical “U” was determined, based on the statistical one-tailed test where a 95 percent confidence 
criteria or better was required to correctly reject the null hypothesis (Hq). Hq assumed that there was no 
difference between the two groups for the tested variable, therefore the level of significance or alpha level 
required to reject the null hypothesis was chosen to be a = 0.05. The experiment was planned for an 
evaluation by 30 pilots with relatively similar flight experience. All pilots were Embry-Riddle 
Aeronautical University (ERAU) instructor pilots with a median flight experience of approximately 
1300 flight hours. Each possessed commercial, multi-engine, and instrument ratings, but none had any 
specific experience in actual icing conditions or any prior training in aircraft handling and upset recovery 
due to a hazardous icing condition. The 30 pilots were randomly divided into a control group (baseline 
flight displays and no ICEPro system) and an experimental group (displays modified with ICEPro cues 
and messages). A scheduling problem resulted in a total of 29 pilot evaluations and unequal sample sizes 
between the control and experimental groups (n^ == 14 and n^- 15). This disparity was accounted for by 
the MWU methodology as provided for in Reference 18. 

9.2 ICEFTD Simulator Description and Capability 

The aircraft model and subsystem components for the ICEFTD simulator were assembled in BAR’s 
commercial off the shelf (COTS) proprietary simulator development environment, D-Six. The simulator 
provided highly representative flight characteristics of the Twin Otter icing research aircraft. Although 
originally designed as a pilot training device, the ICEFTD’ s ability to accurately represent aircraft icing 
characteristics made it a very useful research tool with several advantageous features. One is its capability 
for accurately representing flight characteristics beyond stall AO A, which permits pilots to experience 
post-stall gyrations and tail stall hard-over’s that can occur during upset conditions. Another is the pitch 
control system, which can provide control force gradients and deflections that closely match those 
recorded from flight tests of the NASA Twin Otter during upset events. Lastly, an “aircraft emulator” was 
designed and integrated in order to represent the same sensor feedback and output response as that of the 
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NASA Twin Otter research sensors. Figure 12 is an overview of the ICEFTD as modified for this test. 
The items in red font are those which support ICEPro operation. 

The D-Six simulator also provided excellent three-screen graphics of a visual scene, which was an 
essential element of the final approach and missed approach task. A video recording camera system was 
also employed for each pilot evaluation run in the event a replay of the pilot’s actions needed to he 
observed. During testing, the evaluation pilot wore a headset and was completely enclosed in the 
simulator hy a system of curtains as shown in Eigure 13 and Eigure 14. This isolated the pilot from any 
outside disturbances or distractions, and enhanced the fidelity of the simulation environment. 


CONTROL 

STATION 


ICEPRO 



Figure 12. — ICEFTD Configuration for Test and Evaluation of ICEPro 



Figure 13. — Evaiuation Piiot Station 
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A member of the test team operated the simulator and performed data acquisition and processing 
tasks from a remote engineering station. Consequently, this person could not see the ICEFTD instrument 
panel. Therefore, a copy of the PFD displayed to the pilot was provided as shown in Figure 14. 

A second member communicated with the pilot while acting as test conductor and air traffic control 
(ATC) as shown in Figure 15. 



Figure 14. — Engineering Station 



Figure 15. — Air Traffic Control Station 
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9.3 Pre-Test Pilot Proficiency Training 


Before performing the evaluation tasks for data collection, each pilot was subjected to approximately 
1.5 hr of flight training in the ICEFTD. Pilots were trained for the evaluation task using the respective 
flight displays depending on whether they were in the control or experimental group. Training began by 
having each pilot execute the evaluation approach task profile as shown in Figure 16 in a no-ice or 
baseline configuration under instrument meteorological conditions (IMC). The IMC condition for training 
and testing was set at 400 ft and 1 nautical mile (n mi) visibility. The white boxes in Figure 16 describe 
the flight condition and events during the task and the green shaded boxes show the commands given by 
the test conductor, who acted as ATC at various stages during its execution. The missed approach 
procedure (MAP) with an engine failure was initiated by the test conductor at the approximate position as 
shown in red font. Pilots practiced the approach task according to the standard operation procedures 
(SOP), which specified speeds, configurations, and performance standards throughout the profile. This 
phase of training was important in order for the test team to determine that each evaluation pilot 
possessed the requisite instrument flight skills to satisfactorily perform the task. 

The next phase of training was conducted up and away under visual meteorological conditions (VMC) 
to familiarize pilots and develop their skill in dealing with aircraft characteristics due to the ice protection 
failure case condition. This training emphasized recognition of impending upset conditions and the effects 
of wing flap configuration and flight speed on aircraft handling and control characteristics. Pilots in the 
control group were taught how to most effectively use their baseline flight displays to control the aircraft 
and recognize and recover from upset conditions. Pilots in the experimental group received the same 
training emphasis, but were taught how to integrate their scan and control strategy with envelope protection 
cues from ICFPro to facilitate safe flight control and upset recoveries. During this training, evaluation pilots 
were also instructed to fly with both hands on the yoke. This was a learning experience because nearly all of 
them were used to one handed operation in aircraft with relatively light control forces. 

Once the up and away training was completed, each of the pilots practiced the approach procedure 
with the hazardous failure case icing condition as would be performed in the actual evaluation. A certain 
amount of time was also spent having the pilots practice the missed approach procedure under iced 
conditions. This was the most difficult task to perform because of high pitch control forces and the 
requirement to null the asymmetric thrust with high rudder forces when an engine was failed. Pilots had to 
practice this maneuver several times before they could consistently accomplish it in a proficient manner 
and not lose control. Evaluation pilots were continually reminded to fly according to the SOP, which was 
intentionally designed to place the aircraft near the edges of the flight envelope at all times. This protocol 
forced considerable attention to precise aircraft control throughout the approach task, which was essential 
for making a good utility assessment of ICFPro. The evaluation profile approach task for both groups 
required considerable mental concentration, physical strength and very aggressive control inputs to 
prevent or recover from upset conditions. All pilots, regardless of size or gender, were able to effectively 
perform the approach task where pitch control forces reached 90 to 100 Ibf at times. In general, most 
pilots only required one practice approach before they were ready to begin the evaluation phase. 

In addition to the flight training, pilots were briefed on and practiced completing the NASA TFX 
workload form. Following this training, pilots were critiqued by the test team, and then given a short 
break before they began their evaluation runs. 
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Figure 16. — Evaluation Profile Approach Task Flown by Each Pilot Group During ICEPro Testing and Evaluation 


9.4 Simulator Flight Evaluations 

The evaluation phase required that a total of three satisfactory approach tasks he performed hy each 
pilot. The test conductor sat behind the ICEFTD and acted as ATC and was able to view the evaluation 
pilot through a small opening in the rear curtain. During each evaluation run, the pilots were completely 
enclosed in the simulator cab, and no coaching or extraneous discussions were permitted. As shown in 
Figure 16, the approach task consisted of “radar vectors” to intercept a precision approach procedure. The 
pilot would execute the approach according the SOP, and “break out” of the clouds at 400 ft above 
ground (AGF), and continue the descent to the runway on a visual glide path. At approximately 100 ft 
AGF, “ATC” would instruct a missed approach procedure, which the pilot would acknowledge and then 
initiate. As soon as both power levers were fully advanced, the test conductor would fail an engine, and 
the pilot would continue the procedure while raising the wing flaps from 30° to 10°. The test was 
terminated when the evaluation pilot was stabilized on the missed approach heading. 

Once complete, the evaluation pilot would fill out an electronic TLX form, which automatically 
computed the numerical workload score, and received a short debriefing from the test team. The pilot 
would then re-enter the simulator for the next evaluation run. 

The entire evaluation process for each pilot including the pre-test training exercises and three 
evaluation runs took approximately 3 hr, which seemed to be the limit before fatigue became a factor in 
pilot performance. After testing was completed for each pilot, they were asked to fill out a post-test 
survey questionnaire, included in Appendix B. This questionnaire consisted of four sections using a 
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Likert (Ref. 19) style format: Part I provided pilot demographic and experience information; Part II asked 
about situational awareness; Part III asked for the experimental group’s assessment of ICEPro integration; 
and Part IV asked general questions about workload factors. 

9.5 Data Acquisition and Handling 

Pilot flight performance data parameters, which were available through a flight data file that was 
resident in D-Six, were sampled at 10 Hz. MATLAB routines were used to collect and reduce these data 
for further analysis. For the final approach segment of the task, the Theil inequality coefficient was used 
to evaluate how closely the pilot’s flight path and airspeed agreed with that of a perfectly flown approach. 
A Theil inequality coefficient was calculated for measured lateral and vertical deviations from the 
centerline of the approach path, and the airspeed error from the required 75 kt indicated airspeed (KIAS) 
approach speed. MATLAB routines were also used to record the number of times the stick shaker and 
pusher fired, and number and depth of tail stall AOA exceedence events over the duration of the entire 
flight task. In addition to these performance data, other parameters were recorded during each run to 
provide a more complete picture of the pilot’ s control strategy which was useful in training and debriefing 
the evaluation pilots. These included time histories of pressure altitude {Hp), pitch attitude (9), pitch rate 
iq), normal acceleration (uz), elevator position (S^,), elevator control force (F^), and wing flap setting (8/r). 
Pilots completed the NASA TLX immediately after each data run, and then completed the survey 
questionnaire at the termination of the evaluation session. 

The pilot performance data quantified how well the pilot flew the approach task, how many stick 
shaker and pusher events occurred, and the number of times and extent to which the tail stall AOA was 
exceeded by 5° or more. It should be noted that reaching a tail stall AOA, which was one of the measured 
parameters, did not always result in an upset as long as the pilot immediately recognized the condition 
and promptly arrested the negative pitch rate. The added 5° margin took into account pilot reaction time 
and aircraft dynamic response to the controls. Therefore, tail stall upsets were defined by the number of 
times the pilot reached or exceeded the 5° margin, and the maximum negative AOA achieved in the 
event. This latter metric was also used to define “upset risk”, since the deeper the stall, the lower 
likelihood of recovery. In addition to pilot performance data, the numerical scores of the TLX workload 
assessments and the numerical results from the Likert-type post-test questionnaires were analyzed by the 
MWU process. The TLX analysis was of interest to understand if the added information of the ICEPro 
displays adversely affected workload. The post-test survey analysis was used to assess if there was a real 
and positive difference in hazardous aircraft state awareness, and perception of workload between the two 
groups. Additionally, a section of the survey provided noncomparative information on how well the 
experimental group felt the ICEPro was integrated with flight displays. 

The utility evaluation of ICEPro was based upon assessing real differences between the control and 
experimental groups. The MWU analysis process began by rank ordering the scores of each group, and 
computing a “U” value based on the rank sums. In this one directional test, the “U” for the experimental 
group had to be greater than 144 to ensure a 95 percent probability for correctly rejecting Hq. A 
cumulative probability p was calculated from the z score and was compared to a = 0.05, the requirement 
for accepting the alternate hypothesis. If this probability was less than 0.05, the null hypothesis was 
rejected. A negative z score (number of standard deviations) of more than magnitude 1.69 (i.e., less than 
-1.69) was required to produce the cumulative probability which met this requirement. The parameters 
which were assessed included all quantified measures of pilot performance, TLX workload, and 
subjective opinions from the survey questionnaires. 

An assessment of recovery from tail stall upsets was also performed. In many cases, when a tail stall 
event was encountered, a series of pitch oscillations would ensue from which the pilot would have to 
recover. An analysis of repeated events was performed to determine if the experimental group could 
recover from these oscillations using ICEPro cueing sooner than the control group, and thereby be less 
prone to entering an out-of-control condition. A pilot coupling event (PCE) represents a hazardous 
situation, which if not arrested quickly, could progress to a loss of control. In this context, the risk of an 
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unrecoverable upset was associated with the number of repeated tail stalls in a PCE scenario, and the 
depth of each stall beyond the 5° margin previously discussed. A time window analysis was therefore 
used to analyze the number of tail stalls occurring within a continuous 5, 10, 15, and 20 sec span, while 
the pilot was attempting a recovery. Search scripts in MATLAB were used to analyze the beginning and 
ending of each PCE, and total the number and severity of each tail stall within the PCE. The results were 
then reported in a tabular format for both groups, and analyzed for significance by the MWU method. An 
important outcome of this analysis was to assess the utility of ICEPro AOA cues for minimizing the 
recovery time from a PCE, and thereby reduce risk of a loss of control situation. 

10.0 Pilot in the Loop Results and Discussion 

10.1 Pilot Demographics 

All evaluation pilots were instructor pilots in the ERAU professional pilot training program. Thirty 
pilot volunteers were broken down into two groups — a control group who flew baseline displays, and an 
experimental group who flew with ICEPro modified displays. All evaluation pilots held FAA commercial 
licenses with multi-engine and instrument ratings. The demographic data was collected in Part I of the 
post-test survey questionnaire, and is discussed below. Due to a scheduling problem, the control group 
consisted of one less pilot that the experimental group. The demographics of the groups were as follows: 

Subjects and gender : 

14 control group (12 male, 2 female) 

15 experimental group (12 male, 3 female) 

Median jlight hours : 

Control group Experimental group 

Total 1350 Total 1250 

Multi-engine 122 Multi-engine 100 

Icing related experience and training as related from Part I of the post- test survey questionnaire: 

• 75 percent had no prior in-flight icing experience 

• 89 percent did not feel that their prior icing related knowledge/experience would have prepared 
them to perform the test scenario (training was required in order for pilots to execute the flight 
task) 

• 72 percent agreed that the NASA videos/web based training gave more information about icing 
than they had known before (viewing NASA web-based training videos was a pre-test 
requirement) 

• Mixed responses were given for icing related flight training being focused on IPS operation (this 
depended on the types and classes of aircraft in which pilots had been previously trained) 

10.2 Pilot Control Performance 

Pilot control performance assessment from the control and experimental groups are summarized in 
Table 2. Results highlighted in green indicate that the significance or real difference criterion for the 
assessed parameter was met, and the performance of the experimental group was better than that of the 
control group. Results highlighted in blue indicate that the assessed parameter showed better performance 
by the experimental group, but the result did not meet the real difference criterion. Results highlighted in 
orange, indicate that there was no difference between the performance of the two groups, and results 
highlighted in magenta indicate that the experimental group did not perform as well as the control group. 

In general the cumulative pilot performance for the two most critical events, pusher activation and tail 
stall, met the real difference criterion. These events were considered most important from a safety of 
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flight standpoint because a wing or tail stall upset had occurred, and potential for loss of control was very 
high unless the pilot promptly applied the correct sequence of recovery controls. The shaker activation 
event indicated that the control group had slightly better performance than the experimental group, but 
this finding can be misleading without considering important aspects of shaker implementation and pilot 
use of this functionality in a human factors context. This is explained in the following discussion of the 
results, which were tabulated in Table 2. The first column of Table 2 identifies the event that was 
evaluated during the approach and missed approach task, the second column indicates the number of 
standard deviations (z) required in meeting the real difference criterion, the third column indicated the 
cumulative probability (p), and the last column provides a performance assessment between the 
experimental and control group based upon the analysis. 

10.2.1 Shaker Events — Stall Warning 

The results of the pilot performance for avoiding stick shaker events as shown in Table 2 indicate 
that, cumulatively, the control group had a slightly better performance than the experimental group. 
Shaker firings typically occurred in banked turns if speed got too low, during positive pitch overshoots 
incited by a tail stall induced PCE, and when pilots made an overly aggressive recovery after stick pusher 
firing. The following discussion places these results into perspective and discusses factors that may have 
affected their outcome. The control group flew with basic aircraft displays and a simple stick shaker 
functionality, which always provided a 7 percent stall speed margin based upon a clean wing; i.e., no-ice. 
Therefore, when the shaker fired during the approach task in the icing condition, the actual margin to stall 
was less than 7 percent. 

On the other hand, the experimental group flew with ICEPro generated displays, which provided real 
time AOA cues and shaker functionality that maintained a 7 percent stall speed margin for the icing 


TABLE 2.— SUMMARY OF PILOT CONTROL PERFORMANCE 


Pilot control 
performance by run 
number and 
cumulative 

z score: 

z< -1.69 for 95% 
confidence in Hq rejection 

Probability: 

p < 0.05 required to satisfy a 
(alternate hypothesis) 

Group performance: 

Exp. versus control 
Positive real difference (PRD), 
Better (B), Worse (W), Same (S) 

Shaker events 

Run 1 


1.0474 

0.8575 

W 

Run 2 


-0.2837 

0.3799 

B 

Run 3 


-0.5892 

0.2705 

B 

Cumulative runs 

0.0873 

0.5434 

S 

Pusher events 

Run 1 


-0.0655 

0.4652 

S 

Run 2 


-1.2875 

0.0952 

B 

Run 3 


-2.1822 

0.0137 

PRD 

Cumulative runs 

-1.7457 

0.0385 

PRD 

Stall Messages 

No stall messages appeared for the experimental group 

Tail stall events 

Run 1 


-1.2657 

0.0989 

B 

Run 2 


-1.8330 

0.0318 

PRD 

Run 3 


-1.7894 

0.0350 

PRD 

Cumulative runs 

-2.1822 

0.0137 

PRD 

Legend 


Meets 95% confidence for rejecting Hq 


Experimental group better over control group but <95% confidence for rejecting Hq 


No difference between experimental and control group 


Experimental group performance worse than control group 
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condition. Experimental group pilots could avoid shaker events if they remained helow the upper AOA 
limit bracket as shown in Figure 1 . The control group had no AOA information on their basic flight 
displays. In order to avoid shaker activation, they had to closely maintain the SOP required airspeeds 
throughout the entire task. Notably, the control group was technically at greater risk when the shaker fired 
because their stall speed margin was less than 7 percent. A study of the frequency of shaker events is 
helpful in order to better understand the results of Table 2. Table 3 provides the number of recorded 
shaker events per run for each group of pilots. 

During run one, 14 control group pilots experienced 87 shaker events, whereas 15 experimental group 
pilots experienced 113 shaker events. On average, the control group experienced 6.21 events per pilot, 
and the experimental group experienced 7.53 events per pilot. The corresponding MWU analysis for run 
one in Table 2 clearly shows better (but not significant) performance from the control group. During the 
subsequent runs, however, the experimental group’s performance got better, and this is reflected in the 
analysis for runs two and three in Table 2. If all three runs are considered cumulatively, the normalized 
ratio between the two groups is slightly better for the experimental group, but the cumulative results of 
the statistical analysis shown in Table 2 slightly favors the performance of the control group. Regardless, 
the differences between control and experimental groups as shown in Table 2 and Table 3 are very small. 
What is important to observe here is that the experimental group did a much better job of avoiding shaker 
events during subsequent runs, while the control group’s performance remained nearly constant. This was 
likely due to increased proficiency in using ICEPro AOA cues, which was not fully realized during pre- 
test training. Unfortunately, practical limits on pre-test training were necessary in order to conduct all 
required evaluations within time and program cost constraints. 

10.2.2 Pusher Events — Wing Stall Upsets 

Table 4 summarizes the number of pusher events per run for each group. Pusher events were 
generally precipitated by the same factors that incited shaker events and occurred when the pilot could not 
arrest an AOA increase, which resulted in a wing stall upset. This was particularly evident when the 
missed approach procedure was executed. Here, pilots immediately advanced the throttles to full power, 
raised the wing flaps to 10° and the nose to the takeoff attitude in one coordinated motion. As soon the 
power levers reached the full power stop and the evaluation pilot reached for the wing flap lever, the left 
engine was failed by the test conductor, and the pilot then had to null the yaw and adjust pitch so as to 
maintain a 76 KIAS climb speed. This was a very dynamic maneuver, and in many cases it precipitated a 
PCE. As in the case of the shaker events, the experimental group of pilots improved with each evaluation 
run, whereas the control group actually got worse. The raw data in Table 4 supports the statistical 
assessment that the ICEPro displays resulted in better pilot control performance and lower risk of an upset 
due to either wing or tail stall. 


TABLE 3.— NORMALIZED SHAKER EVENTS 


Control group (14 pilots) 

Experimental group (15 pilots) 

Run 

Shaker events 

Normalized 

Run 

Shaker events 

Normalized 

1 

87 

6.21 

1 

113 

7.53 

2 

92 

6.57 

2 

86 

5.73 

3 

87 

6.21 

3 

72 

4.80 

Cum 

266 

6.33 

Cum 

271 

6.02 


TABLE 4.— NORMALIZED PUSHER EVENTS 


Control group (14 pilots) 

Experimental group (15 pilots) 

Run 

Pusher events 

Normalized 

Run 

Pusher events 

Normalized 

1 

12 

0.86 

1 

13 

0.87 

2 

16 

1.14 

2 

10 

0.67 

3 

19 

1.16 

3 

6 

0.40 

Cum 

47 

1.05 

Cum 

29 

0.64 
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10.2.3 Stall Messages 

A “STALL” message was incorporated into the PFD in the event the pilot attempted to override the 
pusher and thereby remain in a stall upset condition. This was a functionality associated with ICEPro 
displays, and appeared to he unnecessary as all the experimental pilots promptly applied recovery controls 
to reduce the AOA helow the upper har when the pusher fired. No stall messages appeared during any of 
the runs flown hy the experimental group. 

10.2.4 Tail-Stall Upsets and Recoveries 

Tail stall upsets resulted when pilots flew too fast, exceeding the speeds specified at various segments 
of the flight task, or when they allowed the AOA to get too low. This condition was greatly aggravated 
with increasing wing flap deflections, and was most sensitive during the final approach and missed 
approach procedure when the flaps were 30° down. As descrihed previously, a tail stall was defined as 
reaching a negative AOA that was 5° helow that where the tail had technically reached a stall condition. 
This margin was determined experimentally, as it appeared that if a pilot immediately applied recovery 
controls before reaching this negative AOA, recovery would be prompt and the chance of a PCE 
minimized. When a tail stall occurs in an aircraft equipped with mechanical controls, such as the one 
simulated in this test, the pilot feels an immediate increase in pull force and notes a rapid build-up in 
negative pitch rate. In order to arrest the condition, the pilot must quickly exert very high pull forces (on 
the order of 90 tolOO Ibf) to reverse the downward motion of the nose. If successful, the nose starts back 
up; but, as it does, pitch forces immediately lighten and a positive pitch overshoot can easily occur. This 
can precipitate a PCE, which can develop into a loss of control situation should a negative AOA be 
reached that the pilot is not able to physically overcome. Pre-test pilot training acquainted both control 
and experimental group pilots with these characteristics, and the basic methods for recovery. 

During training, pilots practiced tail stall upset recoveries by keeping the pitch divergence to a 
minimum. Additionally, the experimental group was taught to use the ICEPro displays to facilitate 
recovery by adjusting pitch during a PCE to remain within the safe envelope defined by the AOA 
brackets. When pitch rate was negative, the pilot would attempt to pull the nose up, but remain below the 
high AOA bracket. If the pitch rate was positive, the pilot would push the nose down, but remain above 
the low AOA bracket. If done properly, this would result is a very expeditious recovery — normally within 
one or two cycles. Of note, this strategy contributed to fewer pusher events as the experimental group 
tended to be better at avoiding stall AOA than the control group. 

The learning bias evident in the shaker and pusher data seem to indicate that had more time been 
available for the experimental group to practice with the ICEPro displays, their performance during these 
events would have been better. Runs 2, 3, and cumulative of the tail stall event clearly show positive real 
difference in the performance of the experimental group and provide better than 95 percent confidence for 
correctly rejecting Hq. 

10.3 Pilot Precision Approach Performance 

The final approach segment required the pilots to intercept and accurately fly a precision approach to the 
minimum descent altitude of 200 ft. As shown in Eigure 1 , the flight displays for both groups did not 
provide computed steering or a flight director format. Therefore, the pilots had to fly what is termed “raw 
data” and null localizer and glide slope errors by controlling heading and descent rate. As described earlier, 
this task was also performed in IMC until reaching the ceiling of 400 ft above ground level. When the pilots 
“broke out” of the cloud at this altitude, they continued their descent to 100 ft and were directed to execute a 
missed approach. The difficulty in flying the approach task was largely due to control aspects discussed 
earlier. Some pilots were very smooth and precise throughout the task, and were able to execute it without 
too much difficulty. This was because they did not allow the attitude of the aircraft to vary much beyond 
that required to maintain a given flight condition. Generally, these were pilots who had very efficient 
instrument scans, and who made very small but immediate corrections when required. Pilots whose 
instrument scans were less efficient and allowed large pitch or roll variations to occur were more prone to 
get into a pitch axis PCE. A PCE event greatly affected glide slope, course, and airspeed errors, and was 
reflected in the Theil inequality coefficient values for each of those parameters. Table 5 provides the results 
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of the MWU analysis of the precision approach errors between the control and experimental groups. It 
should he noted, that both groups of pilots generally flew the approach procedure within Airline Transport 
Pilot (ATP) standards (Ref. 20) as required in the SOP. The Theil inequality coefficient values, which were 
used in the MWU analysis, were basically a means of quantifying the pilot’s performance with respect to a 
perfectly flown approach. A representative localizer Theil inequality value for one such run is shown in 
Figure 17. Similar plots were obtained for glide slope Theil inequality values. As shown in Table 5, the 
control group had better cumulative performance in minimizing localizer and airspeed errors. The 
experimental group had better cumulative performance in minimizing glide slope error, but did not meet the 
significance criteria required to infer that there was a real difference between the two groups. It would 
appear, after considering the results from Table 2 and Table 5, that pilots who had the advantage of ICEPro 
displays tended to prioritize aircraft control over the preciseness at which they flew the localizer and the 
required airspeed. This is perhaps why the control group, who flew more precise airspeed, had significantly 
more tail stall upset events than the experimental group and was considered to be at greater risk of entering 
an out of control situation. 


Localizer /THEIL Inequality = 0.65 



TABLE 5.— PILOT PRECISION APPROACH PERFORMANCE 


Pilot precision approach 
performance by mn 
number and cumulative 

z score: 

z< -1.69 for 95% 
confidence in Hq rejection 

Probability: 
p < 0.05 required to 
satisfy a (alternate 
hypothesis) 

Group Performance: 

Exp. versus Control 
Positive real difference (PRD), Better 
(B), Worse (W), Same (S) 

Localizer 

Run 1 

0.6328 

0.7436 

W 

Run 2 

0.2619 

0.6116 

W 

Run 3 

0.0436 

0.5260 

W 

Cumulative 

0.1528 

0.5692 

W 

Glide slope 

Run 1 

-1.4402 

0.0718 

B 

Run 2 

0.5674 

0.7221 

W 

Run 3 

-1.4621 

0.0689 

B 

Cumulative 

-0.4364 

0.3233 

B 

Airspeed control 

Run 1 

1.8330 

0.9681 

W 

Run 2 

1.9640 

0.9764 

W 

Run 3 

1.4402 

0.9281 

w 

Cumulative 

1.9421 

0.9752 

w 

Legend 


Meets 95% confidence for rejecting Hq 


Experimental group better over control group but <95% confidence for rejecting Hq 


No difference between experimental and control group 


Experimental group performance worse than control group 
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10.4 Exposure to Risk of an Out of Control Event 


A number of tail stalls, which were encountered in the course of the approach procedure, resulted in 
PCEs before the pilot was able to stop the oscillations and recover to a stable and normal flight condition. 
This event was described earlier in Section 8.5, and under “Tail-stall upsets and recoveries”. Section 
10.2.4. If a quick recovery from a tail stall event (TSE) could not be facilitated by the pilot, there was a 
risk that repeated oscillations result in a PCE which would deteriorate into a LOC situation. Therefore, the 
numbers of repeated tail-stalls during PCEs were summed for each group during the final approach 
segment in four time duration windows of 5, 10, 15 and 20 sec. The methodology began by identifying an 
initial tail-stall event (TSE), which met the criterion (negative tail-stall AOA plus 5°). If another TSE 
occurred within 5 sec it would be counted with the first tail stall and considered a PCE. If in this event, 
TSEs continued to occur within 5 sec of the one previous, they would be added to the PCE until the 
duration of time between a repeat TSE was greater than 5 sec. This methodology was then repeated for 
time intervals of 10, 15, and 20 sec. The exposure to risk of reaching a LOC situation was therefore 
greater during a PCE based upon the number of repeat TSEs in any of the four time frames, the amount of 
time it took the pilot to recover, and the maximum negative pitch attitude reached during the event. The 
results of this analysis are provided in Table 6. 

The most significant events occurred during the 5 sec interval. By using their ICEPro driven displays, 
the experimental group had 75 percent fewer PCEs, their events were approximately half as long in 
duration, and the maximum nose down attitude during the event was about 3.5° less than that of the 
control group. In effect, the experimental group could arrest a PCE quicker, with fewer oscillations than 
the control group. Most important however, was the maximum depth of the tail stall, which was measured 
by the maximum negative pitch attitude achieved during the recovery attempt. The more negative the 
attitude, the higher the negative column forces, and the more difficult it was for the pilot to arrest the 
negative pitching moment. Though not shown here in the data, some pilots had to use in excess of 90 to 
100 Ibf to arrest the negative pitching moment, which was more likely to cause a positive pitch overshoot, 
and sometimes result in pusher activation. 

As previously mentioned, the test profile was intentionally designed to place pilots on the edge of a 
TSE during the approach, and as a result, most pilots in both groups experienced them, especially when 
momentarily distracted from their instrument cross-check. However, when a TSE occurred and especially 
when it deteriorated into a PCE, the experimental group was able to effectively utilize their AOA cues to 
damp the oscillations more quickly than the control group. They accomplished this recovery while 
minimizing their time at an unsafe negative AOA, and thereby were at less risk of reaching a LOC 
situation. 


TABLE 6.— EXPOSURE TO RISK BASED UPON PILOT COUPLING EVENTS 


Group 

Time interval between 
repeat TSEs 
(sec) 

Total number 
of PCEs 

Mean duration of PCEs 
in the time interval 
(sec) 

Mean max 
negative pitch 

Control 

5 

16 

15.16 

-12.23 

Experimental 

4 

8.66 

-8.8 

Control 

10 

69 

18.54 

-12.06 

Experimental 

29 

17.98 

-11.12 

Control 

15 

84 

23.08 

-12.32 

Experimental 

36 

23.22 

-11.37 

Control 

20 

90 

27.64 

-12.44 

Experimental 

41 

25.85 

-11.23 
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10.5 Workload 


After completing each evaluation run, the pilots were asked to complete a TLX workload assessment 
for the entire task. The flight profile was intentionally designed to place each group of pilots very near the 
edges of the safe flight envelope especially during the final approach and missed approach segments. 
Further, the tasks were flown in IMC, and both the cognitive and physical aspects of workload were quite 
high for both groups, especially when recoveries from TSEs or PCEs were required. Because the 
experimental group had to integrate more information than the control group, there was a concern that 
their workload could be adversely affected. 

The ICEPro displays provided much more information than the basic displays, and had to be 
integrated into the pilot’s scan along with the instrument flying task. The control group did not have the 
additional information and thus had less cognitive workload. An analysis was conducted for each of the 
three runs in both groups. The results shown in Table 7 indicate that overall, the experimental group 
reported less total workload. Run 2 for example, taken by itself did meet the significance criteria and 
indicated that ICEPro displays made a real difference in reduced workload. However, on average for all 
runs, the results did not meet the significance criteria. 

Observation of both groups of pilots indicated that as they flew each successive profile, they became 
more comfortable with the task, and tended to report lower workload. In spite of the lower workload, it 
was also apparent that, by the second run, fatigue began setting in; this affected pilot concentration and 
control performance. The entire evaluation was conducted over an approximate 3 hr period with 
strategically inserted rest breaks, but it was apparent that the performance of the pilots from both groups 
seemed to peak by the second evaluation and decrease on the third run. The results of the post-test survey 
questions, which are discussed in the next section, provide further insight into these findings. 

10.6 Post-Test Survey Results 

The questions in all four parts of the survey solicited pilot opinions to help gain insight of the two 
pilot groups with respect to their icing experience, perception of situation awareness, how well ICEPro 
was implemented, and workload issues associated with the flight task. Descriptive analysis was used for 
responses to Part I — Demographics and Part III — ICEPro Implementation. Parts II — Situation Awareness, 
and Part IV — Workload, were analyzed via the MWU methodology for real differences. The survey 
questions solicited either agreement, or frequency of occurrence. A Likert five-answer format was used 
with numbers 1 to 5 assigned to the answers from left to right. Eor the MWU analysis of Parts II and IV, 
the “direction” of the numbered response for each question, i.e., I to 5, or 5 to 1, was defined to indicate 
better performance. The survey questionnaire is provided in Appendix B for the reader’s reference, and 
histograms of the responses to questions in Parts I, II and IV are provided in Appendix C. Descriptive- 
only assessments of experimental group pilot opinions are provided for Part III. 


TABLE 7.— WORKLOAD ASSESSMENT 


Pilot workload by run 
number and cumulative 

z score: 

z< -1.69 for 95% 
confidence in Ho rejection 

Probability: 
p < 0.05 to satisfy a 
(alternate hypothesis) 

Group Performance: 

Exp. versus Control 
Positive real difference (PRD), 
Better (B), Worse (W), Same (S) 

TLX Workload 

Run 1 


-0.5237 

0.2926 

B 

Run 2 


-1.8985 

0.0274 

PRD 

Run 3 


-1.4621 

0.0689 

B 

Average all runs 

-1.0911 

0.1328 

B 

Legend 


Meets 95% confidence for rejecting Ho 


Experimental group better over control group but < 95% confidence for rejecting Hq 


No difference between experimental and control group 


Control group performance better than experimental group 
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10.6.1 Part I — Demographics 

The two pilot groups were intentionally selected from a relatively homogeneous sample. All pilots 
were essentially General Aviation (GA) instructor pilots who shared common experience with training, 
ratings, and types of aircraft flown. All pilots except one were staff instructors at ERAU and suhscrihed to 
the same operational and flight standards as defined hy the University’s professional pilot training 
program. There was only one outlier in the control group, and that individual’s flight hours were not 
included in the standard deviation calculations as this individual’s flight time skewed the data 
considerably. With the elimination of that pilot’s flight hours, the standard deviation of the control group 
was 1054 hr, and the standard deviation of the experimental group was 698 hr. As shown in Figure C.l 
(Appendix C) nearly all pilots had minimal experience with aircraft icing or icing related training before 
participating in this test. 

10.6.2 Part II — Situation Awareness 

Table 8 and Appendix B and Appendix C summarize the situational awareness responses to the 
questions. 


TABLE 8.— SITUATIONAL AWARENESS POST-TEST QUESTIONNAIRE RESULTS 


Pilot post-test survey questionnaire results 

z score; 

z< -1.69 for 95% 
confidence in Hq 
rejection 

Probability: 
p < 0.05 required 
to satisfy a 
(alternate 
hypothesis) 

Group Performance: Exp. vs. 
Control 

Positive real difference 
(PRD), Better (B), Worse 
(W), Same (S) 

Part II — Situational Awareness 

1 . Flight displays were adequate to determine if 
airframe icing was having an effect on aircraft 
characteristics. 

-3.0987 

0.00008 

PRD 

2. Knew minimum safe speed for a given wing 
flap setting within 5 kt. 

-2.2242 

0.0072 

PRD 

3. Knew how to adjust pitch attitude to avoid a 
wing stall or a tail stall upset. 

-2.5750 

0.0047 

PRD 

4. Knew the wing flap settings for safe rate of 
climh in event of an engine failure. 

-2. 5968 

0.0044 

PRD 

5. Had to rely on aircraft control response to 
determine icing effects on pitch, yaw, and roll. 

-4.4298 

0.0000 

PRD 

6. Confident that the final approach airspeed 
would prevent stall. 

-0.3273 

0.3634 

B 

7. Able to avoid aircraft handling problems or 
tail stall upsets when selecting wing flaps down. 

1.1129 

0.8717 

W 

8. Colored bands on airspeed tape useful to 
safely fly the aircraft during the entire flight. 

-2.7714 

0.0026 

PRD 

9. Relied on stick shaker to prevent inadvertently 
stalling. 

-0.1746 

0.4221 

B 

10. Always knew when approaching a wing or 
tail stall condition. 

-1.8985 

0.0274 

PRD 

Legend 


Meets 95% confidence for rejecting Hq 


Experimental group better over control group but < 95% confidence to reject Hq 


No difference between experimental and control group 


Experimental group performance worse than control group 
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A positive real difference (PRD) indicates that the experimental group had significantly better 
situational awareness than the control group for the issue in question. As shown in Table 8, MWU 
analysis of the responses indicated that a PRD existed between the control and experimental groups with 
respect to questions 1 to 5, 8, and 10. To summarize each of these questions, the experimental group had 
significantly better ability to: 

1. Determine that icing was having an effect on flight characteristics, 

2. Know the minimum safe airspeeds to fly for a given wing flap configuration, 

3. Adjust pitch to avoid wing or tail stalls, 

4. Understand when engine out climb performance was being affected by the wing flap setting, and 

5. To know that icing was affecting pitch, roll and yaw characteristics without relying on aircraft control 

response. 

From the responses to question 8 the experimental group of pilots felt the colored bands on the 
airspeed tape were useful for safely flying the aircraft during the entire flight, while the control group 
did not. The histogram of the responses. Figure C.3 (Appendix C), clearly reflects this difference, but 
the result was somewhat confusing since both groups had the same color bands on their airspeed tape 
(Figure 1), and a real difference was not expected. One explanation however, may have been that the 
ICEPro speed carets, when overlaid on the baseline PFD display speed tape, provided better low and high 
speed awareness in relation to the non-iced aircraft condition. Another possibility is that the experimental 
group misinterpreted the question and thought the “color bands” referred to the ICEPro speed carets. In 
any case, the wording of the question could have been better to ensure that no misinterpretation of the 
nomenclature was made. 

Question 10 indicated that the experimental group felt they had significantly better awareness when 
approaching a flight condition that could lead to a wing or tail stall than the control group. Many of the 
experimental pilots verbally commented during their test runs that the real time computed AOA brackets 
were most useful in that respect. 

The responses to questions 6 and 9 respectively, indicated that the experimental group was more 
confident that their final approach airspeed would avoid wing stall, and that they did not have to rely as 
much on the shaker to avoid stalling. These results, although better for the experimental group, did not 
meet the significance criteria. This better, but not significantly better result may have been due to the fact 
that the experimental group knew that they could safely fly to just below the AOA upper bracket and 
avoid the shaker firing. If the brackets started flashing coincident with the aircraft attitude indication and 
the shaker did fire, they could immediately reduce AOA and avoid stall. The combination of AOA 
brackets and shaker gave the experimental group good low speed and incipient stall awareness. The 
control group on the other hand, had to rely on basic (clean aircraft) shaker settings and the likewise basic 
(clean aircraft) airspeed bands on their FED to avoid stalling. 

Eortunately for the control group, the failure case icing configuration in the simulation model resulted 
in a very small stall speed increase and concurrent stall AOA decrease. As a result, if the shaker fired for 
the control group, or if they flew into the amber stall warning band on the airspeed indicator, they were 
generally able to react with a decreased pitch input in time to avoid stall. Had a more severe ice shape 
been modeled and the stall margin been less, or even negative, the control group responses may have been 
different as they could have stalled before the shaker fired. The responses to question 7 indicated that the 
experimental group felt less able to avoid handling problems or tail stall upsets while extending the wing 
flaps than the control group. Referring to Eigure C.3 (Appendix C), it is apparent that the greatest number 
of responses from the control group agreed that they could manage their control inputs to avoid handling 
problems, while there were an equivalent number of disagreement responses from the experimental 
group. This result was somewhat puzzling because it was apparent from the data in Table 6 that 
quantitatively, the experimental group was significantly better than the control group in avoiding tail 
stalls. One possible explanation is that the experimental group was provided with displays that clearly 
showed the effects of wing flap extension on the safe AOA envelope, but were required by the test 
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protocol to ignore those indications and extend the wing flaps regardless. This could have led to their 
response choice because they expected handling problems as wing flaps were extended, and knew that 
they were unavoidable. The question may have been poorly worded in that respect. 

10.6.3 Part III — ICEPro integration 

The experimental group was asked 15 questions regarding the integration of ICEPro displays. Since 
these questions only affected that group, a descriptive statistical approach was taken in the analysis of the 
data. The primary goal of these survey questions was to understand if the implementation of real time 
state assessment into flyable pilot cues did provide utility to the pilot. The key findings that follow list the 
percentage of responses on either side of a “neutral” opinion. 

Figure 18 and Figure 19 represent the agreed / disagreed and always / frequently and infrequently / 
never responses recorded for the experimental group, respectively. 


■ Agreed ■ Disagreed 


1. Messages were not useful and cluttered the 

display . . 

2. Combining visual, aural and tactile cues was 

important 

3. AOA brackets were useful to fly safely 


4. It was difficult to understand relation between 
AOA brackets and Hi/Lo airspeed carets 

5. Synoptic page was more useful than messages for 
state assessment 

7. De-cluttering did not affect ability to recover from 

stall 

8. the "red" CLM and FLP LIM needed less flap setting 

for engine out performance. 

9. "Control buzzing" made aircraft control difficult . . . 


10. Baseline airspeed tapes and carets were 
confusing . . . 

15. Immediately noticed when CLM and FLAP LIM 
messages were on at the same time 



Figure 18. — Agreed versus Disagreed survey responses 
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■ Always/Frequently ■ Infrequently/Never 

0 10 20 30 40 50 60 70 80 90 100 

6. Favored using airspeed carets over AOA brackets 
for safe flight 

11. Flew in middle of AOA brackets 


12. Felt that AOA brackets immediately captured 
attention when flashing 

13. Felt airspeed cues were helpful for maneuvering/ 
. approach 

14. Felt lower AOA bracket helped avoid tail stall 
upset .... 


Figure 19. — Additional survey responses 

Summarizing these key findings, the experimental group felt that the AOA brackets were very useful 
in all phases of flight, and were effectively able to use the brackets to keep the aircraft in a safe flight 
envelope. However, many had difficulty understanding the relationship between AOA and airspeed. None 
of the participating pilots had experience with flying AOA prior to this test, and it is felt that this response 
was largely the result of this lack of experience. 

More pilots felt that presenting both the baseline airspeed along with the ICEPro generated airspeed 
carets was confusing. The original design intent of the ICEPro displays was to present the computed real 
time speeds as “advisory” information. This way, the pilot would always have a reference to the baseline 
aircraft, which would prove useful if a system anomaly was encountered. There was a concern for the 
manner in which this would be done to avoid confusion and the survey seemed to bear this concern out. 
Along those same lines, pilots were essentially ambivalent as to the usefulness of the ICEPro speed carets 
when maneuvering and flying the approach task as shown in Eigure 16. A majority of pilots felt that the 
integration of visual, aural, and tactile cues was important, but providing redundant messages on two 
displays was not necessary. 

Lastly, one major concern was the assessment of the multi-axis control excitation or “buzzing”, which 
was commanded by the ICEPro logic for aircraft state assessment. Some pilots were bothered by the 
buzzing, and some were not. It appeared that because it was expected, pilots accommodated to this as they 
would if having to fly through turbulence. In any case, there did not seem to be a strong opinion one way 
or the other regarding control buzzing. 

10.6.4 Part IV— Workload 

Workload assessments via the NASA TLX format were gathered after each run for each pilot. Post- 
test questionnaires were solicited in an attempt to understand some of the important factors that 
influenced the scores that pilots gave to workload assessment. The survey answers from the control and 
experimental was assessed via the MWU methodology. Table 9 provides the results from the survey 
questionnaire. Pilot workload was not measurably affected by ICEPro cueing. The data indicated the 
following: 

1 . Controlling vertical speed on glide slope was more difficult for experimental group (60 versus 

43 percent). It appears that the control group prioritized safe flight over strict airspeed control. 

2. Plight control inputs were more physically demanding for the control group (73 versus 60 percent), 

possibly due to the fact that the control group was not aware of the magnitude of AOA divergence. 
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3. The control group reported that it took more concentration and effort for flying MAP safely (87 
versus 80 percent). The results however did not meet the significance criterion, hut did show that the 
experimental group did hetter. 

4. Staying on glide slope without pitch upset was more difficult for control group (79 versus 67 percent). 
This was likely because the control group made corrections without knowledge of AOA limits. On 
the other hand, the experimental group had a real time display of AOA limits and could remain within 
those limits while correcting to the glide path. By “respecting” these limits, the experimental group 
tended to give greater priority to remaining in a safe flight conditions and were less aggressive in 
maintaining a precise vertical flight path. It should he noted however, that both groups flew the 
approaches within ATP standards as previously discussed. 

5. Maintaining glide slope detracted from localizer control. This affected the experimental group more 
than the control group (64 versus 47 percent). This is thought to be a result of AOA cueing, which 
captured the pilot’s attention. 

In summary, the workload was not measurably affected by the addition of ICEPro system. Pilots 
seemed to accommodate to those displays rather quickly during their training, and appeared to use them 
quite effectively to perform the flight task. TLX data showed clearly that the flight task was very 
demanding for both groups of pilots, and the physical difficulty of it far outweighed other workload 
dimensions. 


TABLE 9.— PILOT WORKLOAD SURVEY RESULTS 


Pilot post-test survey 
questionnaire results 

z score: 

z< -1.69 for 95% 
confidence in Hq rejection 

Probability: 
p < 0.05 required to 
satisfy a (alternate 
hypothesis) 

Exp. versus Control 
Positive real difference 
(PRD), Better (B), Worse 
(W), Same (S) 

Part IV — Workload 

1 . It was difficult to control vertical speed 
when on glide slope. 

0.7856 

0.7902 

W 

2. Operating the flight controls to avoid 
an upset condition when on final 
approach was a physically demanding 
effort. 

-0.5455 

0.2852 

B 

3. 1 felt it took a considerable amount of 
concentration and effort to safely execute 
the missed approach procedure. 

-1.2002 

0.1108 

B 

4. Keeping the aircraft on the glide slope 
without experiencing a pitch upset was a 
very demanding task. 

-0.8074 

0.2034 

B 

5. On final approach I spent so much time 
trying to fly the glide slope that I was 
unable to maintain good localizer course 
control. 

0.5674 

0.7221 

W 

Legend 

’’■F 

Meets 95% confidence for rejecting Hq 


Experimental group better over control group but < 95% confidence for rejecting Hq 


No difference between experimental and control group 


Control group performance better than experimental group 
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11.0 Atmospheric Turbulence Test Methodology 

The pilot-in-the-loop simulations could not duplicate all of the real-world conditions that could have 
first-order effects on system performance, such as those due to real atmospheric turhulence. Therefore, in 
order to assess system performance under these conditions and minimize the technical risk of an eventual 
flight-test validation program, the effects of atmospheric turhulence on ICEPro were studied. This led to a 
one year extension awarded under NASA’s Vehicle Systems Safety Technologies (VSST) project. This 
section summarizes the study which is fully reported in the Year 5 progress report that can he found in 
Appendix D. 

11.1 Experiment Design 

Four tasks and their measures of success (MOS) were defined in the ICEPro atmospheric turhulence 
test plan (Appendix E). These tasks involved studying the source of the scatter in RT-PID estimates in 
turhulence, understanding the effect of sensor noise and turhulence on RT-PID and D-ICES, determining 
the performance characteristics of a turhulence prohe, and studying the effect of turhulence in different 
phases of flight. In order to test each task as descrihed helow, a test matrix was developed. The test matrix 
included representative flight phases in terms of airspeeds, attitude, and configuration. Utilizing D-Six 
scripting capabilities, a series of scripts were developed to explore the test matrix through hatch testing. 

11.2 Task Descriptions 

This summary will descrihe the questions asked in each of the four tasks. For more detail, please see 
Appendix D. 

11.2.1 Task 1: Simulating Real World RT-PID Performance in Turbulence 

Determine whether the scatter in the turhulence estimates arise from, 1) colored noise in the 
regressor(s), and if so what is the source of the colored noise and in which regressor(s), 2) insufficient 
model structure, and if so how should the model structure he adjusted or, 3) some other means. In the 
system identification process, regressor(s) are defined as the independent variables. 

11.2.2 Task 2: Effect of Sensor Noise and Turbulence 

Modify, test, and validate the use of RT-PID and D-ICES algorithms with atmospheric turbulence as 
an input for various levels of atmospheric turbulence, and to identify an acceptable error in the wind 
measurement during turbulence for RT-PID and D-ICES algorithms. Additionally, verify MOS during 
straight and level flight for varying atmospheric turbulence levels and icing conditions. 

11.2.3 Task 3: Effect of Turbulence for Different Flight Phases at Selected Noise Level 

Modify, test, and validate the use of RT-PID and D-ICES algorithms with atmospheric turbulence as 
an input. Select an acceptable atmospheric turbulence measurement error based on findings from Task 2 
and commercially available atmospheric turbulence sensors. Verify MOS during each phase of flight for 
varying atmospheric turbulence levels and icing conditions. 

11.2.4 Task 4: Evaluate the Effectiveness of Manual Pilot Inputs Alone for Making High 
Confidence Vehicle State Estimates When Performing Typical Maneuvering Flight Tasks 
Under Varying Levels of Atmospheric Turbulence 

Determine the portion or portions of the maneuvers, which provide frequency content at the natural 
frequency of the short period, dutch-roll, and roll modes and thereby support the achievement of high 
confidence parameter estimates from RT-PID. This effort was not performed due to time and funding 
constraints. 
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11.3 Measures of Success 

To ensure that test objectives are attained, measures of success (MOS) have been formulated. A MOS 
is defined as a comparative requirement which must be satisfied by the results of testing. The overall 
measure of success for the primary and secondary objectives will be that the error bounds and accuracy in 
RT-PID and Theil inequality coefficients in D-ICES algorithms meets or exceeds its non-atmospheric 
turbulence performance in atmospheric turbulence. This success criterion focuses on the performance of 
these two algorithms and not on ICEPro’s utility to mitigate a potentially hazardous icing encounter or on 
the pilot’s performance during such encounter. It assumes that if the RT-PID and D-ICES algorithms 
perform equally in and out of atmospheric turbulence, the ICEPro system will perform equally. The 
following MOS are defined for the primary objective. 

RT-PID Algorithm 

1 . Message accuracy 

Message error will be determined by the error bounds and accuracy of its associated stability and 
control coefficient. Eor example, the red (warning level) pitch degrade logic is a function of elevator 
control power 

CtnSe — 0.25 • CmSe _ _nO’ice 

The elevator control power must meet or exceed its non-atmospheric turbulence error bound and 
accuracy in atmospheric turbulence. This is repeated for all ICEPro messages. 

2. ISP accuracy 

ISP error will be determined by comparing the true ISP value with the estimated ISP value. ISP is a 
function of five stability and control coefficients: Longitudinal static stability, elevator control power, 
dihedral effect, aileron control power, and normal lift coefficient. The true ISP value is known since 
the five true stability and control parameters are known to the simulation. The ISP error must meet or 
exceed its non-atmospheric turbulence error bound and accuracy in atmospheric turbulence. 

D-ICES Algorithm 
1 . Control deflection accuracy 

Control deflection error is determined by comparing the estimated deflection Theil inequality 
coefficient with the baseline no turbulence case. The control deflection error must meet or exceed its 
non-atmospheric turbulence error in atmospheric turbulence. 

11.4 Changes to ICEPro Logic 

Several specific features were added to the ICEPro module for the current effort. The new features 
included adding several input parameters within the D-Six variable list. These parameters permitted 
overriding ICEPro’s logic in the Monitor mode forcing the system into Identification mode. In addition, 
user specified surface buzzing times were created to override normal ICEPro buzzing logic. Buzzing 
refers to exercising the controls through a set of orthogonal multi-sine inputs to provide additional control 
content for RT-PID operation. Einally, the use of these new features was optional allowing ICEPro to 
operate in an unmodified logic set. 

Utilizing D-Six scripting capabilities, a series of scripts were developed to facilitate batch testing. 
These scripts interfaced an Excel file containing the test matrix, set up the simulation initial conditions, 
stepped through the simulation using control inputs provided by virtual pilot logic contained in the 
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scripts, recorded selected time history variables, stopped the simulation, and exported the recorded data to 
MATLAB. The script virtual pilot logic used the differences between the desired flight path angle, 
heading, and airspeed and the current actual values as inputs to the elevator, aileron, and throttles as 
control inputs. Sideslip feedback to the rudder was also used to coordinate turns. Equations (11) show the 
basic control laws used by the virtual pilot. The gains were set to low values that worked satisfactorily for 
the no turbulence cases, but were less than optimal with turbulence. These values were used for all cases 
to minimize excessive control movements during turbulent conditions. For some cases, the throttle 
commands were set to zero to eliminate excessive throttle movements encountered with turbulent 
conditions. 


S^command ^y(y desired y)^“ ^yY j"(y desired 

(lla) 

^^command “ ^\|/ desired “ 

(lib) 

^^command 

(11c) 

^Throttle “ (^desired 

(lid) 


11.5 Atmospheric Turbulence Probe 

In support of Task 3, a search was conducted to identify candidate COTS turbulence probes. Several 
probes were identified using state of the art LIDAR technology. Unfortunately, the accuracy of such 
technology requires an order of magnitude improvement for the present work. The Atmospheric 
Integrated Meteorological Measurement System AIMMS-20 probe by Aventech Research was chosen 
because 1) It provides the most accurate wind determination out of the candidate probes and 2) it is 
already part of GRC’s inventory. Table D. 1 in Appendix D summarizes the AIMMS-20 performance 
based on an error analysis. 

12.0 Atmospheric Turbulence Results and Discussion 

A detailed discussion of Tasks 1 to 3 results can be found in the Year 5 progress report (Appendix D). 
The following results and discussion provide a short summary. 

12.1 Task 1 Simulating Real World RT-PID Performance in Turbulence 

Several practical issues related to real-time parameter estimation for a linear longitudinal dynamics 
model in atmospheric turbulence using indirect turbulence measurements were examined and discussed. 
The frequency response of the airflow vanes, wing response to atmospheric turbulence, and the structural 
response of the air data boom were identified as sources of colored noise. The frequency response of the 
airflow vanes for a wide variety of combinations of natural frequency and damping ratio did not adversely 
impact parameter estimation results in turbulence, except for Cz^ and Czg^,. This was attributed to noisy 
explanatory variables in the atmospheric turbulence implementation and relatively low aerodynamic 
contributions as compared to Cz„. Airflow vane dynamics do not substantially contribute to the scatter in 
the estimates, because all of the added dynamics are outside the range of frequencies analyzed. The 
structural modes of the boom lead to similar conclusions, because the natural frequencies of the boom 
structural modes are outside the range of frequencies used for dynamic modeling. 

A major source of colored noise was identified as the frequency-dependent upwash and time delay 
induced by the wing-bound vortex system and the longitudinal separation between the AOA measurement 
and the wing. It was also surmised that the frequency-dependent sidewash was a major source of colored 
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noise. The frequency-dependent upwash and time delay appear to be significant contributors to biasing 
and increased scatter and uncertainty for parameter estimates in atmospheric turbulence. 

Practical issues were examined using data from a Twin Otter DHC-6 longitudinal linear simulation, 
with realistic noise sequences added to the computed aircraft responses. This allowed a clear view of the 
effect of each source of colored noise to the modeling problem, because the true values of the model 
parameters were known. This approach was used initially to show that real-time parameter estimation can 
be done accurately in all atmospheric turbulence conditions if the AOA and angle of sideslip 
measurements are accurate. Flight test data from the GRC DHC-6 Twin Otter aircraft was used to validate 
the effect of the identified colored noise sources. 

Based on these findings, several practical recommendations for flight testing in atmospheric 
turbulence are suggested, as well as areas for future study. First, mount the air data boom at the nose of 
the aircraft to minimize the effect of the wing-bound and wing-tip vortices. Second, if the air data boom is 
mounted on the nose of the aircraft, an upwash and time delay calibration as a function of frequency is 
required. Third, airflow angle measurements must be corrected for frequency dependent upwash, 
sidewash, and time delay corrections. 

The following Task 2 and Task 3 results were carried out assuming that these precautions had been 
taken care of. This assumed that the frequency response was adequate (Good assumption based on the 
results of this section for a well calibrated, stiff, nose mounted sensor). This left the sensor’s static 
performance to be evaluated in Task 2 and Task 3. In other words, if these precautions are taken, the RT- 
PID estimates will behave similarly inside and outside of atmospheric turbulence. 

12.2 Task 2: Effect of Sensor Noise and Turbulence 
RT-PID 

• In general, the original MOS for message accuracy were not achieved since the errors in the 
estimates and in the error bounds increase as a function of turbulence intensity. A new MOS 
criterion was formulated where the error in the estimates and in the error bounds with 
atmospheric turbulence is tolerated as long as the ICEPro messages and AOA brackets are 
equivalently displayed inside and outside of turbulence. 

• In practice, for flight in zero, light, and moderate atmospheric turbulence the average airflow 
angles are similar. In severe turbulence, the average airflow angles are significantly different due 
to wing stall. The differences do not affect message accuracy, but do affect the AOA bracket 
accuracy. The data from stalled runs was not considered useful because stalled flight does not 
produce reliable data. As a result, all severe turbulence test conditions have been omitted from 
Task 2 analysis. 

• With the exception of the roll degrade message in calm atmosphere, there were no false positives 
or false negatives. The roll degrade message was caused by low pilot activity resulting in poor 
RT-PID estimates. In each of the turbulence cases, the additional aileron inputs provided by the 
virtual pilot improved the RT-PID estimates so that no false positives (roll degrade message) 
were annunciated. 

• In general the new MOS message criterion was considered to be successful. 

D-ICES 

• It was found that the application of turbulence and sensor noise on D-ICES had minimal impact 
on performance without filtering. 

o Only one condition was found that yielded a false positive that would have caused the system 
to enter ID Mode when no icing was present. 

o There was one condition where the elevator deflection Theil inequality Coefficient for an iced 
airplane failed to exceed the threshold and two where the value originally exceeded the 
threshold, but with time decreased until it was below the threshold. 
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• However, when the state vectors were low pass filtered to remove higher frequency disturbances 
due to turbulence and noise, D-ICES accurately detected all icing conditions in the presence of 
turbulence and sensor noise. Filtering did not have any adverse ramifications on logic, thresholds, 
or other system parameters or functions. 

• Three potential false positive cases were found that showed the elevator Theil Coefficient built up 
over time to exceed the threshold when no ice was present. An examination of the data in this 
region led to supposition that the D-ICES inversion process had trouble determining the correct 
elevator deflection with pitch due to elevator characteristics at those flight conditions. This 
condition is specific to the Twin Otter flight model. 

Using the results of Task 2, it was determined that the objective to identify an acceptable error in the 
wind measurement was not required and that the performance of an air data probe such as the AIMMS-20 
probe was satisfactory. 

12.3 Task 3 Effect of Turbulence for Different Flight Phases at Selected Noise Level 

For the purpose of this task the four phases of flight are defined as: 

Phase 1. Flaps up straight and level flight at 1 10 KIAS and 2500 ft - 30 sec duration 
Phase 2. Flaps up straight and level flight at 1 10 KIAS and 2500 ft followed by a constant altitude flap 
transition (0° to 20°) and deceleration to 80 KIAS - 1 min duration 
Phase 3. Heading capture at 80 KIAS, 2500 ft, and flaps 20° from 090° to 360° - 30 sec duration 
Phase 4. Descent from 3000 ft at 75 KIAS and flaps 20° followed by a flap transition (30°) and 
deceleration to 70 KIAS with localizer and glide slope tracking - 2 min duration 

RT-PID 

• Phase 1 runs had no pitch degrade, climb limit, or flap limit messages for zero, light and moderate 
turbulence levels. The MOS criterion for AO A bracket was applied to Phase 1 and considered to 
be successful. 

• Phase 2 runs had no pitch degrade, yaw degrade, climb limit, or flap limit messages for zero, 
light, and moderate turbulence levels. The MOS criterion for AOA bracket was applied to Phase 
2 and considered to be successful. 

• Phase 3 runs had no pitch degrade, climb limit, or flap limit messages for zero, light and moderate 
turbulence levels. The MOS criterion for AOA bracket was applied to Phase 3 and considered to 
be successful. 

• Phase 4 runs had no pitch degrade or yaw degrade messages for zero, light or moderate 
turbulence levels. 

• The MOS criterion for AOA bracket was applied to Phase 4: Applying the MOS criterion would 
be unsuccessful in this phase since a pilot is able to discern the unconservative variation in the 
AOA brackets. It is recommended that additional piloted runs be conducted to verify this 
conclusion by the virtual pilot. 


D-ICES 

• Phase 1 runs were a repeat of the Task 2 conditions and the results were similar. 

• Phase 2 runs experienced no false positives during this scenario. 

• In Phase 3 runs, a false negative was seen for the iced airplane with no turbulence, but correctly 
predicted positives for the low and moderate turbulence levels. The false positive was believed to 
be due to a singular condition where there was little difference between the iced and clean 
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airplane’s control effectiveness at the chosen flight condition. This condition is specific to the 
Twin Otter flight model. 

• In Phase 4 runs, there were no false indications, even though the results remained close to a 
singular condition seen in the level turning flight case. However, the Theil inequality coefficient 
values are still sufficient to exceed the threshold for transitioning to ID mode for all of the iced 
conditions. 

• In most cases, D-ICES performed as designed in the presence of turbulence and noise. However, 
the exceptions noted above, especially the false negatives, do indicate that some further 
development of the system is warranted. One possible area to examine would be utilizing other 
terms related to the Theil inequality coefficient, namely the bias, variance and covariance terms in 
some way to develop a more robust detection scheme. 

• In general, it was found that the performance of the AIMMS-20 probe and the new MOS criterion 
were satisfactory for Task 3. 

13.0 Conclusions and Recommendations 

13.1 ICEPro System Design and Implementation 

In keeping with the NASA goal to improve the safety of commercial and general aviation aircraft, an 
envelope protection system was developed using no direct ice detection devices. Through the use of 
measured data, a priori information, and real-time stability and control estimates, this system computes an 
icing severity parameter. This parameter is used to trigger appropriate pilot cueing to warn the pilot of 
hazardous conditions and provide guidance for action to mitigate the situation. The main advantage of the 
system is that it uses data that are readily available in modern avionics systems and once adjusted to 
specific aircraft makes and models, and mated to modern display systems, should provide sufficient pilot 
cueing for envelope protection against icing related incidents. 

13.2 Pilot in the Loop Test and Evaluation of ICEPro 

The utility of the Icing Contamination Envelope Protection (ICEPro) system for mitigating a 
potentially hazardous icing condition was evaluated by 29 pilots using the ICEFTD. The pilots 
participating in this test were divided into two groups; a control group using baseline displays without 
ICEPro, and an experimental group using ICEPro driven display cueing. Each group flew identical 
precision approach and missed approach procedures with a simulated ice protection failure icing 
configuration. Pilot performance, workload, and survey questionnaires were collected for both groups of 
pilots. Results showed that real time assessment cues were effective in reducing the number of potentially 
hazardous upset events and in lessening exposure to loss of control following an incipient upset condition. 
Pilot workload with the added ICEPro displays was not measurably affected, but pilot opinion surveys 
showed that real time cueing greatly improved their situation awareness of a hazardous aircraft state. The 
conclusions from this test indicate that the experimental group of pilots, who flew with ICEPro displays, 
demonstrated significantly better control performance in dealing with aircraft handling problems under 
icing conditions and had better awareness of a hazardous aircraft state. Pilot workload was not affected by 
the additional display information provided by ICEPro, but pilot opinions of the implementation of that 
system indicate that improvements could be made. 

13.3 Atmospheric Turbulence and Sensor Noise 

The conclusions from this task confirm the hypothesis that the scatter in the estimates observed 
during flight in turbulence is due to airflow angle colored noise. Several important sources of colored 
noise were identified and tested. A major source of colored noise was identified as the frequency- 
dependent upwash and time delay induced by the wing-bound vortex system and the longitudinal 
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separation between the AOA measurement and the wing. It was also surmised that the frequency- 
dependent side wash was a major source of colored noise. The frequency-dependent upwash and time 
delay appear to be significant contributors to biasing and increased scatter and uncertainty for parameter 
estimates in atmospheric turbulence. 

These findings also disproved the hypothesis that additional model structure was needed. 

Additionally, several precautions should be taken for improved estimates in atmospheric turbulence. 

1) Mount the flow angle sensors as far forward of the wing as practical, preferably on the nose of the 
aircraft. This will minimize airframe flex and the amplitude of the upwash correction. 2) Calibrate the 
change in upwash and sidewash correction versus frequency. 3) Eliminate the use of booms for flow 
angle sensors. This will minimize boom flex. In general, it was found that the performance of the 
AIMMS-20 probe was satisfactory for Task 2 and Task 3. Acceptable noise error was therefore defined 
by current specification of AIMMS-20 probe. As long as booms are not used and the calibration for wing 
response to turbulence has been taken care of, ICEPro performed similarly inside and outside of 
atmospheric turbulence. 

Subsequent analysis of the RT-PID and D-ICES algorithms in Task 2 and Task 3 demonstrated good 
performance with few exceptions. Additional recommendations have been made to look at the exceptions. 

14.0 Recommendations 

1 . Plight displays in this test were not optimized from a human factors standpoint, nor were the envelope 
protection cues coupled with a flight director. Based upon the opinions extracted from the pilot 
survey, pilot control performance and state awareness may be improved by addressing those issues in 
future pilot in the loop testing. 

2. Execute Task 4 in the one year VSST extension. This task evaluates the effectiveness of manual pilot 
inputs alone for making high confidence vehicle state estimates when performing typical 
maneuvering flight tasks under varying levels of atmospheric turbulence. 

3. Plight test to validate RT-PID findings based on colored AOA regressor findings. 

4. Flight test to validate ICEPro. 
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Appendix A. — Abbreviations and Symbols List 

Abbreviations 

AOA angle of attack 

D-ICES Dynamic Inversion Control Evaluation System 

ERAU Embry-Riddle Aeronautical University 

ECD Elight Control Display 

Ho null hypothesis 

ICEETD Ice Contamination Effects Flight Training Device 

ICEPro Icing Contamination Envelope Protection system 

IPS Ice Protection System 

ISP Icing Severity Parameter 

J cost function 

LSE least-squared error 

PCE Pilot Coupling Event 

RT-PID Real Time Parameter Identification 

PFD Primary Flight Display 

RMS root-mean square 

SLD Super-cooled Large Droplet 


Symbols 


A stability matrix 

B control matrix 

C stability matrix 

C/p rolling moment due to sideslip angle derivative, 5C//5P 

C/g_j rolling moment due to aileron deflection derivative, dCijdba 

Ci 5 r rolling moment due to rudder deflection derivative, dCi/ddr 

Cip rolling moment due to nondimensional roll rate derivative, dCi jd^pb / 2U) 

C/^ rolling moment due to nondimensional yaw rate derivative, 8Ci ld{rbl2V ) 

C,„a pitching moment due to AOA derivative, 8Cm /5a 

C,„gg pitching moment due to elevator deflection derivative, 8Cml8he 

C,„q pitching moment due to nondimensional pitch rate derivative, 8C,n / 8{qc /2V) 

CNa normal force due to AOA derivative, 8Cn j 8a 

CN 5 e normal force due to elevator deflection derivative, 5C;v /5Sg 

C^q normal force due to nondimensional pitch rate derivative, 8Cn / 8{qc /2V) 

C„p yawing moment due to sideslip angle derivative, 8Cnl8f> 

Cn^a yawing moment due to aileron deflection derivative, 8Cnl85a 

Cn^r yawing moment due to rudder deflection derivative, 8Cnl8hr 
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Cnp yawing moment due to nondimensional roll rate derivative, dCn ld{pb 1 2V) 

Cnr yawing moment due to nondimensional yaw rate derivative, 8Cn / d{rb / ) 

Cyp side force due to sideslip angle derivative, dCy jd^ 

Cy^a side force due to aileron deflection derivative, BCy jdba 

Cy^r side force due to rudder deflection derivative, dCy/ddr 

Cyp side force due to nondimensional roll rate derivative, dCy /d[pb/2V) 

Cy^ side force due to nondimensional yaw rate derivative, dCy /d{rb /2V) 

U Theil inequality coefficient 

ajc row of matrix A 

bk row of matrix B 

b reference span 

c mean aerodynamic chord 

t time 

u control vector 

w control vector predicted from inversion routine 

uk control 

V true velocity, ft/s 

X state vector 

xO state vector initial condition 

xk state 

y, y measured and predicted output 

a alternate hypothesis 

CD frequency, rad/s 

9 parameter vector 

0 estimated Parameter 

^ confidence hound 
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Appendix B. — Post-Test Survey Questionnaire 


Part I. Demographics 

Name: MaleCZI Female CZI 

Pilot Number: 

Total Flight Hours: 

Multi - Engine Hours: 

FAA Ratings: Commercial Multi|^ CFI CFII 

Aircraft Flown and Hours in Type: 


Icing related training and experience: 

1. In my all my prior flying experience, I encountered in-flight icing 

Never Rarely Sometimes Very often Always 


2. 1 felt that my prior icing related knowledge and experience would have adequately prepared me to 
perform this test scenario without the familiarization training I received on aircraft handling 
characteristics. 

Strongly Disagree Strongly Agree Disagree Undecided Agree 

3. The NASA videos and web based training materials provided me with information about icing that I 
had never known about before. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 


4. Before this test, my icing related flight training was mostly focused on how to operate the ice 
protection system. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 
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Part II. Situation Awareness 


1 . My flight displays were adequate for me to determine when airframe icing was having an effect on 
aircraft characteristics. 

Never Rarely Sometimes Very often Always 

2. 1 felt as though I knew the minimum safe speed I could fly for a given wing flap setting within 5 kt. 

Never Rarely Sometimes Very often Always 

3. 1 knew how to adjust my pitch attitude to avoid a wing stall or a tail stall upset. 

Never Rarely Sometimes Very Often Always 

4. 1 wasn’t always sure which wing flap settings would allow a safe rate of climh in the event of an engine 
failure. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

5. 1 relied solely on aircraft control response to determine how icing affected pitch, yaw, or roll 
characteristics. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

6. 1 was confident that the final approach airspeed I chose to fly after I departed the final approach fix 
would prevent me from stalling. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

7. While I was putting the flaps down during the approach and landing, I felt I could effectively manage 
my control inputs and airspeed to avoid aircraft handling problems or tail stall upsets. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

8. The various colored hands on the airspeed tape were useful for me to safely fly the aircraft during the 
entire flight. 

Never Infrequently Sometimes Frequently Always 
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9. 1 relied upon the stick shaker to prevent me from inadvertently stalling. 


Never Rarely 


Sometimes Very Often Always 


10. 1 could always tell when I was approaching a wing or tail stall condition. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 


Part III, Implementation of Envelope Protection Cueing (Experimental ICEPro Group Only) 

1. 1 felt that it was important to combine the visual cues from ICEPro with aural alerts, and tactile 
feedback from the stick shaker. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 


2. The ROL DGRD, PTCH DGRD, or YAW DGRD messages on the PFD did not provide useful 
information, and only cluttered the display. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

3. When flying at slow speeds, I could easily use the AO A bars as a good pitch control reference for 
safely flying the airplane. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 


4. 1 had difficulty understanding the relationship between the AOA bars and the high and low speed carets 
on the airspeed tape. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 


5. The color coded flight control surfaces on the flight control synoptic were more useful than the 
messages for assessing degraded control state. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

6. 1 favored using the airspeed carets rather than the AOA bars to remain within a safe operating 
condition. 

Never Infrequently Sometimes Frequently Always 
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7. When I unintentionally got into an upset condition, the disappearance of all ICEPro messages except 
the AOA bars on the PFD did not affect my ability to recover the airplane. (Answer only if a wing stall or 
tail stall upset occurred). 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

8. When I extended the wing flaps and the red CLIMB LIM and FLAP LIM messages came on together, I 
immediately knew if an engine failed I would have to reduce my flap setting in order to climb. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

9. The occasional “buzzing” of the flight controls by ICEPro made flying the aircraft very difficult. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

10. 1 tended to confuse baseline airspeed limits with the airspeed limits (carets) that were posted by 
ICEPro. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

1 1 . 1 tended to fly so as to keep my pitch attitude in the middle of the AOA bars when they were 
presented by ICEPro 

Never Infrequently Sometimes Frequently Always 

12. When the AOA bars started flashing, they immediately captured my attention. 

Never Infrequently Sometimes Frequently Always 

13.1 felt that the airspeed carets from ICEPro were useful for helping me determine safe flight speeds 
when maneuvering during approach and landing. 

Never Infrequently Sometimes Frequently Always 

14. The low AOA cue on the PFD enabled me to avoid a tail stall upset. 

Never Infrequently Sometimes Frequently Always 
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15. 1 immediately noticed when the CLIMB LIM and FLAP LIM messages were on at the same time. 


Strongly Disagree Disagree Undecided Agree Strongly Agree 

Part IV. Workload 

1. 1 found it difficult to control vertical speed when on glide slope. 

Never Infrequently Sometimes Frequently Always 

2. Operating the flight controls to avoid an upset condition when on final approach was a physically 
demanding effort. 

Never Infrequently Sometimes Frequently Always 

3. 1 felt it took a considerable amount of concentration and effort to safely execute the missed approach 
procedure. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

4. Keeping the aircraft on the glide slope without experiencing a pitch upset was a very demanding task. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

5. On final approach I spent so much time trying to fly the glide slope that I was unahle to maintain good 
localizer course control. 

Never Infrequently Sometimes Frequently Always 
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Response Frequency Response Frequency 


Appendix C. — Results of Survey Questions from Parts I, II, and IV 
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Figure C.1 . — Pilot demographics — Part I Survey questions 1 to 4 
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‘Direction of responses that indicated better situational awareness 
Figure C.2. — Situational Awareness — Part II Survey questions 1 to 5 
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Response Frequency Response Frequency 
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Figure C.3. — Situational Awareness — Part II Survey questions 6 to 10 
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Figure C.4. — Workload — Part IV Survey questions 1 to 5 
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Appendix D. — One Year Atmospheric Turbulence Extension Final 
Report — Effect of Turbulence and Sensor Noise On ICEPro 


Progress Report: Year 5 


Borja Martos, Principal Investigator 
University of Tennessee Space Institute (UTSI) 
411 B. H. Goethert Pkwy 
Tullahoma, TN 37388-9700 

and 


Ryan Oilman, David Gingras, and Billy Barnhart 
Bihrle Applied Research (BAR) 

81 Research Drive, 

Hampton VA 23666 


Cooperative Agreement: NNX07AD58A 
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List of Symbols and Abbreviations 


Abbreviations 

AIAA 

American Institute of Aeronautics and Astronautics 

AIMMS-20 

Atmospheric Integrated Meteorological Measurement System-20 

AOA 

angle of attack 

ATP 

Airline Transport Pilot 

BAR 

Bihrle Applied Research 

COTS 

commercial off the shelf 

D-ICES 

Dynamic Inversion Control Evaluation System 

D-Six 

Bihrle Applied Research’s Simulation Environment Software 

10 

no ice condition 

12 

20 min ice accumulation condition 

ICEFTD 

Ice Contamination Effects Flight Training Device 

ICEPro 

Icing Contamination Envelope Protection System 

ID 

identification 

ISP 

icing severity parameter 

IVHM 

Integrated Vehicle Health Monitoring 

KIAS 

knots indicated airspeed 

ELDAR 

light detection and ranging 

MIL 

military handbook 

MOS 

measure of success 

NASA 

National Aeronautics and Space Administration 

GRC 

NASA Glenn Research Center 

LaRC 

NASA Langley Research Center 

NASA TLX 

NASA Task Load Index 

OBES 

Onboard Excitation System 

PFD 

Primary Flight Display 

PELF 

power for level flight 

RMS 

root mean square 

ROC 

rate of climb 

RT-PID 

Real Time Parameter Identification 

SFTE 

Society of Flight Test Engineers 

TO 

no turbulence condition 

TI 

light turbulence condition 

T2 

moderate turbulence condition 

T3 

severe turbulence condition 

TMS 

Turbulence Model Structure 

TOGA 

Take Off - Go Around 

UTSI 

University of Tennessee Space Institute 

VIAS 

velocity indicated airspeed 

VMC 

visual meteorological conditions 

VSST 

Vehicle Systems Safety Technology 

Symbols 

Az 

longitudinal acceleration 

A, 

lateral acceleration 


vertical acceleration 


NASA/CR— 2014-218320 


54 


C/p 

C/5a 


CmaQ 

Cmas 

Cma 

Cmo 

Cmq 

Cm5e 

CW(x 

deU 


g 

//p 


mean aerodynamic chord 

dihedral effect stability coefficient 

aileron effectiveness stability coefficient 

gust longitudinal static stability coefficient 

rigid body longitudinal static stability coefficient 

longitudinal static stability coefficient 

pitch stability coefficient 

pitch damping stability coefficient 

elevator effectiveness stability coefficient 

normal lift coefficient 

rudder effectiveness stability coefficient 

Theil inequality coefficient 

gravitational acceleration constant 

pressure altitude 

aircraft velocity relative to the earth 


Vt true airspeed 


Vw 

Vd 

Ve 

Vn 

Vs 

Vt 

P 

q 

q 

r 

u 

V 

w 

A 

a 

CL/n 

P 

Pm 

5 f 

8. 

0 

it) 

¥ 


true wind velocity vector 
velocity down 
velocity east 
wind velocity, down 
wind velocity, east 
wind velocity, north 
velocity north 
stalling airspeed 
true airspeed 
roll rate 

dynamic pressure 
pitch rate 
yaw rate 

longitudinal velocity 
lateral velocity 
vertical velocity 
change 

angle of attack 
measured angle of attack 
angle of sideslip 
flow sideslip angle 
flap deflection 
elevator deflection 
pitch angle 
roll angle 
yaw angle 
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D.l Abstract 


A team comprised of personnel from The University of Tennessee Space Institute, Bihrle Applied 
Research, Inc., NASA Glenn Research Center (GRC), and NASA Langley Research Center (LaRC), has 
successfully developed a prototype Icing Contamination Envelope Protection System (ICEPro). The ICEPro 
system facilitates flight envelope protection hy making continuous real time vehicle state assessments, 
which are synthesized into flyahle pilot cueing along with visual and aural alerts during in-flight icing 
conditions. The simulation could not duplicate all of the real-world conditions that could have first-order 
effects on system performance, such as those due to real atmospheric turbulence. Therefore, in order to 
assess system performance under these conditions and minimize the technical risk of an eventual flight-test 
validation program, the effects of atmospheric turhulence on ICEPro are studied in this extension. 

This extension focused on the impact turhulence has on the ICEPro system. Turhulence will degrade 
the quality of air data sensing. This study explored the accuracy of ICEPro’ s ice detection and parameter 
estimation with perfect sensing and degraded sensing to determine an acceptahle amount of degradation 
that would allow the ICEPro system to function correctly. Einally, after an acceptahle error in air data 
collection was defined, it would he possible to determine whether the tolerance was obtainable in a 
commercially available air data sensor. 

D.2 Introduction 

D.2.1 Research Background 

D.2.1.1 Previous Studies and Results 

A team comprised of personnel from The University of Tennessee Space Institute (UTSI), Tullahoma, 
Tennessee, Bihrle Applied Research Inc (BAR), Hampton, Virginia, NASA Glenn Research Center (GRC), 
Cleveland, Ohio, and NASA Langley Research Center (LaRC), Hampton, Virginia, has successfully 
developed a prototype Ice Contamination Envelope Protection System (ICEPro). ICEPro was developed 
using a DH-6 Twin Otter simulation for development and evaluation. The aerodynamic math models of the 
Twin Otter were derived from a series of wind tunnel tests for both the clean and iced airplanes. The math 
models were then validated against NASA flight data for both the clean aircraft and the test aircraft flown 
with artificial ice shapes. The Twin Otter was chosen because of the availability of the flight test data. 

The ICEPro system facilitates flight envelope protection by making continuous real time vehicle state 
assessments, which are synthesized into flyahle pilot cueing along with visual and aural alerts during in-flight 
icing conditions. Detection of degraded aircraft stability and control and performance due to icing is carried 
out by a dynamic inversion control evaluation system (D-ICES) that compares expected aircraft response from 
a-priori knowledge base with the current response. When differences reach defined thresholds, real time 
parameter identification (RT-PID) methods are invoked to estimate current state parameters, which 
continuously support pilot cueing and alerts. The development effort included simulation-based design, testing, 
and verification. A pilot in-the-loop study was conducted to gather pilot performance data and opinions of the 
utility of ICEPro during simulated icing encounters. Results of the study indicated that the system performed 
as expected and pilot performance benefited from the envelope protection cues. 

figure D. 1 presents a flow chart of the ICEPro system. (Some aspects of the system such as mode 
latching/delatching and logic for displaying messages, logic to force flushing the D-ICES and RT-PID 
buffers, etc. are not shown for clarity). The system starts in Monitor mode where D-ICES calculates, via 
an inversion technique using an a priori clean aircraft aerodynamic model, what the elevator deflection 
should be for the flight condition. This is compared with the actual elevator deflection using a statistical 
parameter called the Theil inequality coefficient. If this coefficient exceeds its threshold the system 
transitions to the ID mode. In the ID mode, the Real Time Parameter Identification (RT-PID) module is 
started and it predicts, based on flight state parameters, the linearized aerodynamic coefficients. These 
coefficients in conjunction with comparable clean and iced coefficients provided by D-ICES are used to 
calculate an Icing Severity Parameter (ISP). If the ISP becomes larger than its threshold, the system 
transitions to the Report mode where pilot cues and messages are displayed indicating the degraded 
aircraft’s condition and providing limits for safe operation of the aircraft. 
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Figure D.1 . — ICEPro System Flow Chart 

D.2.1.2 Current Year Five Study 

The simulation used for the development and evaluation did not duplicate all of the real-world 
conditions that could have first-order effects on system performance, such as those due to real 
atmospheric turhulence. Therefore, in order to assess system performance under these conditions and 
minimize the technical risk of an eventual flight-test validation program, the effects of atmospheric 
turhulence on ICEPro are studied in this extension. 

This extension focused on the impact turhulence has on the ICEPro system. Turhulence will degrade 
the quality of air data sensing. This study explored the accuracy of ICEPro’ s ice detection and parameter 
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estimation with perfect sensing and degraded sensing to determine an acceptable amount of degradation 
that would allow the ICEPro system to function correctly. Finally, after an acceptable error in air data 
collection was defined, it would be possible to determine whether the tolerance was obtainable in a 
commercially available air data sensor. 

D.2.2 Research Objectives 

Four tasks and their measures of success (MOS) have been defined in the ICEPro atmospheric 
turbulence test plan (see Appendix E). This summary will describe the questions asked in each of the first 
three tasks and a brief introduction into the software tools developed to accomplish this work. The effort 
on Tasks 1 to 3 exhausted the allowed time and remaining funds, therefore no Task 4 results were 
included. 

D.2.2.1 Task 1: Simulating Real World RT-PID Performance in Turbulence 

Determine whether the scatter in the turbulence estimates arise from, 1) colored noise in the 
regressor(s), and if so what is the source of the colored noise and in which regressor(s), 2) insufficient 
model structure, and if so how should the model structure be adjusted or, 3) some other means. In the 
system identification process, regressor(s) are defined as the independent variables. 

D.2.2.2 Task 2: Effect of Sensor Noise and Turbulence 

Modify, test, and validate the use of RT-PID and D-ICES algorithms with atmospheric turbulence as 
an input for various levels of atmospheric turbulence, and to identify an acceptable error in the wind 
measurement during turbulence for RT-PID and D-ICES algorithms. Additionally, verify MOS during 
straight and level flight for varying atmospheric turbulence levels and icing conditions. 

D.2.2.3 Task 3: Effect of Turbulence for Different Eligbt Phases at Selected Noise Level 

Modify, test, and validate the use of RT-PID and D-ICES algorithms with atmospheric turbulence as 
an input. Select an acceptable atmospheric turbulence measurement error based on findings from Task 2 
and commercially available atmospheric turbulence sensors. Verify MOS during each phase of flight for 
varying atmospheric turbulence levels and icing conditions. 

D.2.2.4 Changes to ICEPro Logic 

Several specific features were added to the ICEPro module for the current effort. The new features 
included adding several input parameters within the DSix variable list. These parameters permitted 
overriding ICEPro’ s logic in the Monitor mode forcing the system into identification mode. In addition, 
user specified surface buzzing times were created to override normal ICEPro buzzing logic. Buzzing 
refers to exercising the controls through a set of orthogonal multi-sine inputs to provide additional control 
content for RT-PID operation. Finally, the use of these new features was optional allowing ICEPro to 
operate in an unmodified logic set. 

Utilizing DSix scripting capabilities, a series of scripts were developed to facilitate batch testing. These 
scripts interfaced an Excel file containing the test matrix, set up the simulation initial conditions, stepped 
through the simulation using control inputs provided by virtual pilot logic contained in the scripts, recorded 
selected time history variables stopped the simulation, and, exported the recorded data to MATLAB. The 
script virtual pilot logic used the differences between the desired flight path angle, heading, and airspeed and 
the current actual values as inputs to the elevator, aileron, and throttles as control inputs. Sideslip feedback 
to the rudder was also used to coordinate turns. Equations (D. 1) show the basic control laws used by the 
virtual pilot. The gains were set to low values that worked satisfactorily for the no turbulence cases, but 
were less than optimal with turbulence. These values were used for all cases to minimize excessive control 
movements during turbulent conditions. For some cases, the throttle commands were set to zero to eliminate 
excessive throttle movements encountered with turbulent conditions. 

S^command = desired “ y)+ -^pY + J (y desired ~ j)dt (D.la) 
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(D.lb) 


Srcommand — ^pP 

Sxhrottle = Ky (Vdesired — V) + K^V 


(D.ld) 


(D.lc) 


D.3 Discussion of Study and Results 

There are at least two ways to account for turbulence in system identification algorithms, 1) model the 
turbulence mathematically and estimate the corresponding parameters and 2) measure the turbulence. In the 
first approach, the identification of dynamic models containing turbulence is classified as a system with state 
and measurement noise. In the second approach, favored in this work, turbulence is measured indirectly 
since air turbulence is imbedded in the true wind velocity vector. This approach is favored because it makes 
it possible to use commercial off the shelf sensors to measure the atmospheric turbulence and treat it as a 

measured explanatory variable in the system identification problem. The true wind velocity vector, , of 
atmospheric air relative to the surface of the Earth is found from the vector subtraction: 


Where v, the true aircraft velocity relative to the air mass and is the velocity of the aircraft with respect 

to the Earth. Since is the difference between two large quantities it is essential that and are 
measured as accurately as possible in order to minimize errors in . The details of these calculations 
can be found in flight test engineering handbooks (Refs. 1 to 3) The most obvious difficulty in measuring 
air turbulence from an aircraft is the removal of the aircraft’s velocity with respect to the Earth. 

Additional difficulties arise since the aircraft distorts the airflow around itself. This requires precise 

measurement of Vf and Eg . Any inaccuracies in the measurements or calibrations will affect the derived 
wind, and consequently the aerodynamic estimates. It is only recently that COTS technology has been 
developed with sufficient measurement accuracy. 

A search was conducted to identify candidate COTS atmospheric turbulence sensors. Several sensors 
were identified using state of the art LIDAR technology with an airflow angle accuracy of 1.0°. In 
comparison, flight test air data booms with traditional air data technology are calibrated to an accuracy of 
0.1°. The cutoff accuracy for this work was set at 0.1°. Unfortunately, the accuracy of LIDAR based 
sensors requires an order of magnitude improvement to be useful for the present work. Several additional 
sensors were identified using traditional air data technology. The AIMMS-20 probe by Aventech 
Research was chosen because 1) It provides the most accurate wind determination out of the candidate 
probes and 2) it is already part of GRC’s inventory. Table D.l summarizes the AIMMS-20 performance 
based on an error analysis. 

Turbulence was tested as calm, light, moderate, and severe. Turbulence implementation and levels 
were defined per MIL-E-8785C and were based on the Dryden Turbulence Spectrum. According to MIL- 
E-8785C, the probability of exceeding light turbulence levels is between 10 ' and 10^^, for moderate 
turbulence, the probability is approximately 10^^, and for severe turbulence it is approximately 10^^, as 
shown in Figure D.2 (Ref. 4) derived from MIL-F-8785C. Therefore, severe turbulence levels are not 
very likely to be encountered. The RMS turbulence wind speed values are also shown in Figure D.2 as a 
function of altitude. The turbulence velocity components were generated randomly within the guidelines 
of the Dryden spectrum and added to the aircraft velocity components to generate the total velocity 
components. A random number was used to generate the velocity components from the turbulence 
spectrum. By specifying a constant initial random number seed, the sequence of random numbers become 
reproducible allowing an “apples to apples” comparison of results. This method is referred to as a “fixed” 



(D.2) 
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seed. Likewise, using a varying random number seed results in unique sequences of random numbers 
from test to test. The three velocity components from the turbulence calculations were added to their 
appropriate wind frame velocity components {u, v, w) that were generated from the aircraft dynamics so 
that the velocity components used for the vehicle motion calculations included the components from the 
turbulence model. 


TABLE D.I.— AIMMS-20 PROBE PEREORMANCE 


Parameter 1 Sigma Error 

Longitudinal acceleration, (g) 0.003 

Lateral acceleration, Ay, (g) 0.003 

Normal acceleration, A^, (g) 0.05 

Angle of attack, a, (deg) 0.10 

Angle of sideslip, |3, (deg) 0.10 

Pitch rate, q, (dps) 0.01 

Roll rate, p, (dps) 0.07 

Yaw rate, r, (dps) 0.01 

Pitch attitude, 0, (deg) 0.20 

Roll attitude, (|), (deg) 0.20 

Yaw attitude, v|/, (deg) 0.20 

GPS velocity down, V^, (mps) 0.10 

GPS velocity north, (mps) 0.10 

GPS velocity east, Vp (mps) 0.10 

True airspeed, Vj, (mps) 0.20 

Wind velocity down, VT^, (mps) 0.30 

Wind velocity north, VT^, (mps) 0.30 

Wind velocity east, Wp (mps) 0.30 


Medium/High Altitude Turbulence Intensities (Probability of Exceedance) 



Figure D.2. — Turbulence Definition (derived from MIL-F-8785C). 
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D.3.1 Task 1 


The following results are representative of the effect of turbulence on aircraft system identification. 
Data were collected using NASA’s AIRSTAR T2 model during atmospheric turbulence conditions. 
NASA’s T2 model is a 5.5 percent dynamically-scaled generic transport model. It is outfitted with typical 
flight test instrumentation and air data booms with flow vanes on each wingtip. In Figure D.3, each color 
represents a different run, and only the last estimate of each real-time parameter estimation run is shown. 
A total of nine runs are represented; six in high turbulence and three in low turbulence. 

Simulating the correct cause and effect due to atmospheric turbulence on aircraft parameter 
identification is an ongoing research topic. It is known that colored measurement noise causes scatter and 
bias (Refs. 5 and 6) in the parameter estimates, which is observed in Figure D.3 during repeated flight test 
maneuvers in atmospheric turbulence. However, it is unknown whether the scatter in the turbulence 
estimates arise from 1) colored noise in the regressor, and if so what is the source of the colored noise and 
in which regressor 2) inadequate model structure, and if so how should the model structure be adjusted or 
3) some other means. 

D.3.1. 1 Hypothesis 

Colored AOA regressor noise and/or inadequate model structure is the cause of scatter in aircraft 
system identifcaiton estimates in atmospheric turbulence. 

D.3. 1.2 Sources of Scatter in Estimates 


Based on the hypothesis a literature search was carried out to examine previous related work. The 
results are summarized in the following sections. 

D.3.1.2.1 Colored Regressor Noise 

Based on the literature search, the most likely causes of colored regressor noise are the wing 
aerodynamic response, the frequency response of long air data booms, and the frequency response of 
airflow angle vanes in atmospheric turbulence. 
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Figure D.3. — Effect of Turbulence on Modeling Results 
(Dr. Eugene A. Morelli, personal communication, April 24, 2011) 
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D.3.1.2.1.1 


Flexure of AirData Boom 


During low-level flight tests over the terrestial and marine boundary layer, a Long EZ research 
aircraft was used to measure atmospheric turhulence (Ref. 7). Details of the aircraft and its 
instrumentation system are found in the cited reference. The hoom motion with respect to the center of 
gravity (accelerometers were mounted at each location) is summarized in the following figure (Ref. 7) 
during light atmospheric turhulence. The phase trace compares fuselage and prohe accelerometer sensors, 
whereas the magnitude traces represent each individual accelerometer sensor. 

Figure D.4 illustrates the complexity of motion for this particular hoom-airframe comhination. 
Significant difference is apparent in Figure D.4 between the vertical motion of the probe and the CG. 
Below 0.6 Hz the coherence between the two accelerations is 1.0 with zero phase shift. Between 0.6 Hz 
and 2.0 Hz the coherence decreases and the phase changers from zero to -45°. Beyond 6 Hz there is no 
significant relationship between the two accelerations. In other words, in the frequency range of interest 
for rigid body aircraft dynamics (0. 1 to 2.0 Hz), the drop in coherence and phase indicates that the AOA 
is different between locations. This in turn results in colored noise in the a regressor. No doubt there 
would be similarities for other boom-airframe combinations but with different frequency responses. 

In order to investigate the in-flight structural response of the air data boom, airflow angle flight data 
collected in a DHC-6 Twin Otter during multiple parameter etstimation maneuvers was analyzed in the 
frequecy domain using Fourier coefficients. Figure D.5 shows the frequency response data for the entire 
frequency range, up to the Nyquist frequency at 25 Hz. The longitudinal and lateral natural frequencies for 
the dominant structural mode of the air data boom are approximately 8 and 9 Hz. These natural frequencies 
are associated with the AOA and angle of sideslip measurements respectively. In Figure D.5, the Fourier 
coeffcients for the AOA near 8 Hz are at the noise floor. Therefore, the structural response of the nose boom 
on which the AOA sensor is mounted, and from fuselage bending, is not a source of colored noise in the 
AOA. Similar results were calculated for angle of sideslip but the results are not shown. 



Frequency (Hz) 

Figure D.4. — Comparison between vertical acceierations of the boom 
and airpiane center of gravity 
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Angle of Attack 



Figure D.5. — Fourier coefficients for measured AOA 

D.3.1.2.1.2 Wing Response to Turbulence 

During flight tests in the marine boundary layer, a Twin Otter research aircraft was used to study the 
aerodynamic effects on wind turbulence measurements(Ref. 8). Details of the aircraft and its 
instrumentation system are found in the cited reference. Although mounted on the wing the airdata boom 
was not used as the primary source of air data measurements. Instead a flush air data system was mounted 
on the nose of the Twin Otter. The flush air data system avoids the vibration and frequency response 
problems of a long boom mounted on the nose of the aircraft (Ref. 9). The wing response to atmospheric 
turbulence can be summarized in Figure D.6 during a 30 km flight in light atmospheric turbulence. Figure 
D.6 represents the estimated AOA scale upwash correction versus frequency. 

The frequency dependant upwash correction is caused by chordwise and spanwise upwash varations 
as a result of the wing vortex system response to atmospheric turbulence (Refs. 8 and 9). The frequency 
scale is the reduced frequency kh based on the half wing span where co is the angular frequency, b is the 
wingspan, and U is the airspeed. The AOA upwash correction (1 + k„) ’ is explained by examining a 
sensor model: 


+ (D-3) 

where 

ttg = AOA bias 

- measured AOA 
aj - true AOA 

In Figure D.6, the vertical axis (1 + k„) ’ represents the correction required to calculate the true AOA 
from measured data. 


ar -(l + k„) -ttfi) (D.4) 

At low frequencies (representing flight in a calm atmosphere) the scale factor is approximately 0.75 
and increases to 0.90 as kb approaches 2.0. A kt value of 2.0 (using the DHC-6 Twin Otter wing span and 
1 10 kt cruise airspeed) corresponds to approximately 2.0 Hz. Therefore, over the frequency range of 
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interest for rigid body aircraft dynamics (0.1 to 2.0 Hz), the scale factor changes from 0.75 to 0.90 due to 
atmospheric turbulence. It was also surmised that the frequency-dependent sidewash was a major source 
of colored noise but this was not tested. 

Figure D.7 depicts the time delay as a function of frequency. The time delay is caused by the 
separation between the nose boom mounted airflow vanes (measurement position) and the wing tip (lift 
and upwash generation), along which turbulence eddies travel. 

The magnitude of the time delay represents the correction required to calculate the true AOA from 
measured data. In the frequency range of interest for rigid body aircraft dynamics (0. 1 to 2.0 Hz), the time 
delay is approximately 0. 1 sec. 
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Figure D.6. — Angle of attack upwash versus frequency 
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Figure D.7. — Time delay versus frequency 
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D.3.1.2.1.3 Frequency Response of Airdata Vanes 


Although References 10 to 12 were found to determine the frequency response of airdata vanes via 
natural frequency and damping ratio, no analysis was found confirming that the frequency response of 
airdata vanes was adequate in atmospheric turbulence. 

D.3. 1.2.2 Model Structure 


It is conceivable that the aerodynamic coefficients are different in atmospheric turbulence that in 
smooth air although such differences have not been documented in flight (Ref. 13). The new model 
structure — referred to as the Turbulence Model Structure (TMS) — includes new coefficients and 
Cmag.These parameters represent the nondimensional pitching moment due to (rigid body dynamics) 
and the nondimensional pitching moment due to (atmospheric turbulence) respectively. Similar 
coefficients were implemented for the normal force. These coefficients represent different contributions 
due to the effect of atmospheric turbulence on the aircraft. Stated differently, knowing the effect of 
atmospheric turbulence on a sensor does not imply that the effect of atmospheric turbulence on the 
aircraft response can be calculated by simply multiplying a by the usual a coefficients. It is assumed that 
the effect of turbulence on aircraft response is proportional to the effect of turbulence on the a sensor; 
therefore, turbulence coefficients are estimated using turbulent a as a regressor to capture its effect. So 
it’s not really a turbulent a coefficient as much as it is an effect of turbulence using the best measure of 
turbulence. Both aerodynamic model structures are presented in Section D.3. 1.3.1. 

D.3.1.3 Modeling 

For this study, a simplified Twin Otter aircraft simulation was generated outside the DSIX software 
environment using the MATHWORKS SIMULINK software. The simulation simplified the nonlinear 
equations of motion (Ref. 14) to 3 degrees of freedom (3-DOF) longitudinal dynamics and a linear 
aerodynamics model. The simulation rate was 50 Hz and the pertinent output variables were corrupted 
with gaussian noise with magnitudes that resulted in a signal to noise ratio of 50 for the measured outputs. 
The use of this simplified simulation was warranted to avoid changes to the high fidelity DSIX database 
as model structure changes were investigated. 

D.3.1.3.1 Model Equations 

For reference purposes, a detailed listing of the 3-DOF longitudinal equations of motion (Ref. 14) for 
a symmetric rigid aircraft is included. 

Force Equations: 


u = qw- gsinQ + {X + Tx)/ m 


(D.5a) 


w-qu + g cos Q + {Z + 1^)/ m 


(D.5b) 


Moment Equation: 


lyq^M + Mj 


(D.6) 


Kinematic Equation: 



(D.7) 


Navigation Equations: 


Pyy =MCOs0-t wsin0 


(D.8a) 
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/z = Msin0-wcos0 


(D.8b) 


Atmospheric Turbulence Contributions: 


^ ^aerodynamics ^turbulence 

(D.9a) 

“ '^aerodynamics ''^turbulence 

(D.9b) 

Where the atmospheric turbulence velocity components, and were calculated using 

MIL-E-8785C specification and the Dryden Turbulence SIMULINK block for calm, light, moderate, and 
severe turbulence. Nondimensional aerodynamic force and moment coefficients for an aircraft can be 
computed from flight measurements as follows: 

Cx ={max -Tx)/qS 

(D.lOa) 

Cz={maz -Tz)lqS 

(D.lOb) 

Cm=[h^-^T]IW 

(D.lOc) 


The nondimensional aerodynamic force coefficients are transformed from the body axis to the wind 
axis as follows: 

=-C 2 Cosa + C;if sina (D-11) 

For local modeling over a short time period, the force and moment coefficients computed from 
Equations (D.IO) can be modeled using linear expansions in the aircraft states and controls: 

Ch = Ci^lS.as + (D.12a) 

C,„ = Aa5 + ^ + (D.12b) 

Atmospheric Turbulence Model Structure 

Cl =Cl^^cls +Ci^^AaQ +Ci^^Ade + Ci^ (D.12c) 

Cm — Cm^AoLg + Aclq + Cm^,,A5e + Cmo (D.12d) 

The A notation indicates perturbation from a reference condition. In Equations (D.12b) and (D.12d), 

C, nQ represents the nondimensional pitching moment at a reference condition, and similarly for the other 
expansions. 

D. 3.1.3.2 Wing Response to Turbulence 

In order to investigate the effects of upwash correction as a function of frequency and time delay as a 
function of frequency the recursive Eourier transform in the RT-PID algorithm was modified. The 
recursive Eourier transform is defined as: 

(D.13) 
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Where the frequency vector is defined asf- [0.1:0.04:2.0], the time delay is modeled as: 


_ -7'2n*timedelay 


H/) 


(D.14) 


Where time delay, is a vector of time delays for each frequency/^ [0.1:0.04:2.0]. The upwash correction 
1 + kii(f)is a vector of upwash corrections for each frequency/ = [0.1:0.04:2.0]. The updated recursive 
Fourier transform is defined as: 

= + (D.I5) 

D.3.1.3.3 Vane Sensor Model 

In order to investigate the frequency response of airflow angle vanes, a second-order airflow vane 
sensor dynamic model was studied. A natural frequency of ©„ = 9 Hz and a damping ratio = 0.35 were 
chosen as representative values for a dynamic model of the airflow vanes (Ref. 10 to 12). Typical natural 
frequency and damping ratio values range from 5 to 20 Hz and 0.2 to 0.6, respectively. Equation (D.6) 
represents a second order continous-time transfer function: 


G{s) 
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+ 2C,(i)„s + (i) 
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(D.16) 


The transfer function representation does not support non-zero initial conditions; therefore, the vane 
sensor model was converted to a state space representation which supported a trim a initial condition. 

D.3.1.4 Testing 

The hypothesis was tested using the SIMULINK model descrihed and the following test conditions: 

1. Baseline run with a perfect AO A measurement and the aerodynamic model structure defined in 
Equation (D. 12). 

2. Based on findings in Section D.3.1.3.3, repeat test condition 1 with a representative frequency 
response model for the airflow angle vanes. 

3. Repeat test condition 2 with varying AOA magnitude and time delay as a function of frequency. 

4. Based on findings in Section D.3. 1.2.2, add turbulence coefficients to the aerodynamic model 
structure and a perfect a measurement. 

D.3.1.5 Results and Analysis 

The following parameters are common to all four test conditions. 

1. Aircraft trimmed at 10,000 ft and 140 kt 

2. 20 sec runs 

3. Orthogonal multi-sine inputs (Ref. 14) invoked from 1 to 11 sec 

4. Model parameter estimates taken at the end of each 20 sec run 

5. 200 runs per atmospheric turbulence intensity (calm, light, moderate, and severe) 

6. Eourier transform frequency spacing 0.04 Hz 

7. Eourier transform frequency range [0. 1 2.0] Hz 

8. Signal-to-noise ratio of 50 

9. 50 Hz data rate 
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The following comments are common to all of the figures associated with test conditions one to four: 

1 . Green line — truth value of the parameter 

2. Red dashed line — confidence interval using +10 percent of truth value of the parameter 

3. Red diamond — parameter estimate 

4. Blue bars — estimated two sigma error hounds 

In the dynamic modeling process, no attempt was made at estimating the drag coefficients, because 
the aircraft excitations did not appreciably changed airspeed, resulting in a low signal-to-noise ratio for 
the axial force coefficient parameter estimation. Additionally, contributions from the lift damping 
coefficient Ci^ were small and omitted from the model and the estimation process. In total, five 
coefficients were estimated, Ci^, C^g^, C^a, C,nq, and C,„gg. 

D.3.1.5.1 Test Condition 1 

As shown in Figure D.8, the baseline model estimates for Cza resulted in improved estimates with 
increasing atmospheric turbulence. This is not representative of actual flight data and is attributed to 
perfect a measurement given the additional excitation caused by atmospheric turbulence. 

As shown in Figure D.9, the baseline model estimates for C^g^ resulted in greater scatter as 
turbulence intensity increased. This is caused by noisy explanatory variables (Ref. 15) from the 
atmospheric turbulence implementation in Equation (D.9). Similar patterns were observed with the 
remaining coefficients Czq, Cz^g, and C,„q, although the results are not shown. These simulations 
demonstrate that all if the AOA measurements are perfect, all model parameters can be estimated 
accurately for any level of turbulence. 
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Figure D.8. — Test Condition 1 — Effect on Cz„ 
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Figure D.9. — Test Condition 1 — Effect on Cmge 


D.3.1.5.2 Test Condition 2 

Estimates for Cz„ using the second-order vane sensor model of Equation (D.6) (co„ -9 Viz and 
- 0.35) caused degradations and biasing in the estimates with increasing turbulence intensities, as 
shown in Eigure D. 10. This is representative of actual flight data. 

As shown in Eigure D.l 1, the baseline model estimates for resulted in greater scatter as 
atmospheric turbulence intensity increased. This is caused by noisy explanatory variables (Ref. 15) from 
the atmospheric turbulence implementation in Equation (D.9), and the vane sensor model. Similar 
patterns were observed with the remaining coefficients Cz^, Czg^, Cma, and Cmq, although the results are 
not shown. 
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Figure D.10. — Test Condition 2 — Effect on Cz„ 
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D.3.1.5.3 Test Condition 2 — Expanded Testing 

Although no quantitative analysis was done, qualitatively, small differences were observed between a 
perfect and typical vane frequency response in test conditions 1 and 2. This is expected because the vane 
dynamics do not substantially contribute to the scatter in the estimates, since all of the added dynamics 
are outside the range of frequencies analyzed. 

To further validate this claim, additional testing was carried out using the same simulation. A total of 
121 different combinations of flow angle vane natural frequency and damping ratios were tested. The 
natural frequency values tested were 5 through 15 in 1 Hz increments. The damping ratio values tested 
were 0.01, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, and 0.6. Since the AIMMS-20 probe approaches 
a gain of one and zero phase shift, it is characterized by the results of test condition 1 . The AIMMS-20 
probe offers superior frequency response performance to mechanical vanes by using 1) a 5 hole pressure 
probe , 2) short tube lengths (air data tubes are typically less than 5 in. long), and 3) the resonance 
frequency of the silicon pressure diaphragm is in the kilohertz region. The following figures depict the 
results of 121 combinations, times 4 turbulence levels, times 200 runs per case. This totaled 96,800 runs. 
Figure D.12 depicts the results of 96,800 runs for Cza- As an example, the bottom left corner block in 
severe turbulence is light blue. The color represents the number of exceedances of Cz„ from +10 percent 
of truth. The remaining 120 blocks are dark blue representing less than 10 exceedances. Figure D.12 
suggests that any combination of natural frequency and damping ratio tested provides adequate 
performance. Similar results were obtained for C,„a and C,nq results are not shown. 
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Figure D.12. — Number of exceedances for Cz„ 


NASA/CR— 2014-218320 


71 






Figure D.13 shows that independent of the natural frequency and damping ratio tested, accurate 
estimates of C,„g^ are feasible in atmospheric turhulence. This is expected due to strong elevator control 
authority. 

Results for Czq, not shown, demonstrate that independent of the natural frequency and damping ratio 
tested, exceedances less than 20 out of 200 or +10 percent of truth are not feasible in any level of 
atmospheric turbulence except calm. For the remaining stability and control estimates, the results show 
that most combinations of natural frequency and damping ratio tested provide adequate performance in 
calm, light, moderate, and severe atmospheric turbulence. 

D.3.1.5.4 Test Condition 3 


As shown in Figure D. 14, estimates for Cza resulted in degraded estimates with increasing 
atmospheric turbulence for varying AOA magnitude and time delay as a function of frequency 
implemented using the data in Figure D.6 and Figure D.7. The degradation is seen as biasing in the 
estimates and increased scatter and uncertainty estimate error bounds. This is representative of actual 
flight data. 

As shown in Figure D. 15, the estimates for Czg^ also resulted in degraded estimates with increasing 
atmospheric turbulence. The degradation is seen as biasing in the estimates and increased scatter and 
uncertainty estimates. The increase in biasing and uncertainty compared to Cza estimates is caused by the 
relatively small contribution of the elevator term to the normal force coefficient. 
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Figure D.13. — Number of exceedances for Cmge 
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Figure D.14. — Test condition 3 — Effect on Cz„ 
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Figure D.15. — Test condition 3 — Effect on Czge 
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Figure D.16 shows that the estimates for C,„a degraded with increasing atmospheric turhulence. The 
degradation is seen as biasing in the estimates and increased scatter and uncertainty estimates. This is 
representative of actual flight data. 

As shown in Figure D. 17, the estimates for Cmg degraded with increasing atmospheric turhulence. The 
degradation is seen as biasing in the estimates and increased scatter and uncertainty estimates. The 
estimates for C,„g resulted in the largest biasing of all the aerodynamic estimates considered because 
angular velocity model parameters are most susceptible to time delays (Ref. 16). 

Figure D. 1 8 shows that the estimates for C,„gg degraded with increasing atmospheric turbulence. The 
degradation is seen as a scatter in the estimates and increased scatter and uncertainty estimates. This is 
representative of actual flight data and correlates with the flight results presented in Figure D.3. 

Corrupting the AOA measurement by a frequency-dependent magnitude and time delay affected all of 
the estimated stability and control coefficients. The estimates of C,„a and C,„gg exhibited moderate biasing 
and increased scatter and uncertainty, Cza and C,„q estimates had large biases and increased scatter and 
uncertainty, and Czg^ estimates showed the largest biases and increases in scatter and uncertainty. 

Previous work on the effects of time-shifted data on flight determined stability and control estimates 
(Ref. 16) using the output-error method in the time domain demonstrated similar trends in the biasing and 
increased uncertainty, compared to the results shown here due to atmospheric turbulence. 


Calm Atmosphere 



Run Number 


Moderate Turbulence 


Liglit Turbulence 

0i 1 ^ 



- 2 [ 
-2 5 


• 3 ' ' * ' 

0 50 100 150 200 

Run Number 


Severe Turbulence 



Run Number 


Run Number 


Figure D.16. — Test condition 3 — Effect on Cm„ 
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D.3.1.5.5 


Test Condition 4 


Based on the findings in Sections D. 3. 1.2.2, additional turbulence coefficients were added to the 
model structure. All of the turbulence estimates for except for the estimates in calm atmosphere 
resulted in large degradations. In fact, the degradations were so large that they were not captured by the 
standard plot range. A similar pattern was observed with the estimates of C^a- This is not representative 
of actual flight data. The estimates for resulted in greater scatter and an overall bias as turbulence 
intensity increased. The overall biasing effect is not representative of actual flight data. Similar patterns 
were observed with the remaining coefficients results are not shown. 

D.3.1.6 Task 1 Conclusions 

Several practical issues related to real-time parameter estimation for a linear longitudinal dynamics 
model in atmospheric turbulence using indirect turbulence measurements were examined and discussed. 
The frequency response of the airflow vanes, wing response to atmospheric turbulence, and the structural 
response of the air data boom were identified as sources of colored noise. The frequency response of the 
airflow vanes for a wide variety of combinations of natural frequency and damping ratio did not adversely 
impact parameter estimation results in turbulence, except for Czq and Cz^g- This was attributed to noisy 
explanatory variables in the atmospheric turbulence implementation and relatively low aerodynamic 
contributions as compared to Cza- Airflow vane dynamics do not substantially contribute to the scatter in 
the estimates, because all of the added dynamics are outside the range of frequencies analyzed. The 
structural modes of the boom lead to similar conclusions, because the natural frequencies of the boom 
structural modes are outside the range of frequencies used for dynamic modeling. A major source of 
colored noise was identified as the frequency-dependent upwash and time delay induced by the wing- 
bound vortex system and the longitudinal separation between the AOA measurement and the wing. It was 
also surmised that the frequency-dependent sidewash was a major source of colored noise. The 
frequency-dependent upwash and time delay appear to be significant contributors to biasing and increased 
scatter and uncertainty for parameter estimates in atmospheric turbulence. 

Practical issues were examined using data from a Twin Otter DHC-6 longitudinal linear simulation, 
with realistic noise sequences added to the computed aircraft responses. This allowed a clear view of the 
effect of each source of colored noise to the modeling problem, because the true values of the model 
parameters were known. This approach was used initially to show that real-time parameter estimation can 
be done accurately in all atmospheric turbulence conditions if the AOA measurement is accurate. Flight 
test data from the GRC DHC-6 Twin Otter aircraft was used to validate the effect of the identified colored 
noise sources. 

Based on these findings, several practical recommendations for flight testing in atmospheric 
turbulence are suggested, as well as areas for future study. First, mount the air data boom at the nose of 
the aircraft to minimize the effect of the wing-bound and wing-tip vortices. Second, if the air data boom is 
mounted on the nose of the aircraft, an upwash and time delay calibration as a function of frequency is 
required. 

The following Task 2 and Task 3 results were carried out assuming that these precautions and 
calibrations had been taken care of. This assumed that the frequency response was adequate (good 
assumption based on the results of this section for a well calibrated, stiff, nose mounted sensor). This left 
the sensor’s static performance to be evaluated in Task 2 and Task 3. This eliminated the need to update 
the D-Six Twin Otter simulation from previous studies. In other words, if these precautions are taken, the 
RT-PID estimates will behave similarly inside and outside of atmospheric turbulence. 

D.3.2 Task 2 

Through initial testing it was found that the Twin Otter at a typical cruise condition was not capable 
of controlled flight in severe turbulence with the virtual pilot used for this study. During runs in severe 
turbulence, the aircraft would consistently enter a wing stall. The data from stalled runs was not 
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considered useful, because stalled flight does not produce reliable data. As a result, all severe turbulence 
test conditions have been omitted from Task 2 analysis. Additionally, non-ice test conditions were run, 
but in the context of RT-PID, were considered outside of the scope and omitted from the analysis. Both 
ice and no ice runs were considered when evaluating D-ICES performance. 

The purpose of Task 2 was to modify, test, and validate the use of RT-PID and D-ICES algorithms 
with atmospheric turbulence as an input for various levels of atmospheric turbulence, and to identify an 
acceptable error in the wind measurement during turbulence for RT-PID and D-ICES algorithms. Wind 
measurement error was varied from 0.0 to 1.0 kt. Eighty test runs were conducted with and without ice, 
turbulence, and wind measurement errors. 

D.3.2.1 D ICES Study Results 

It was found that the application of turbulence and sensor noise on D-ICES had a minimal impact on 
performance without filtering. With the filtering scheme of state variables described in Section D. 3. 2. 1.1, 
D-ICES performance allowed accurate detection of possible icing situations in the presence of turbulence 
and sensor noise. D-ICES used a Theil inequality coefficient, comparing the a-priori elevator deflection 
for a clean aircraft to that of the current flight condition, to statistically estimate the aircraft’s current 
state. The estimated clean elevator deflections were calculated using a flight model inversion technique to 
find a corresponding control deflection for the given flight condition. The Theil inequality coefficient 
provided a measure of how well the actual elevator deflection matched the calculated elevator deflection, 
where a value of 0 indicated a total match, and 1 indicated a total mismatch. If the coefficient exceeds a 
threshold of 0. 1 8 with a 3 sec delay to latch true, D-ICES activates the IVHM ID mode. 

As a result, a metric for success with sensor noise and turbulence is determined by their effect on 
creating false positive or false negative mode switches. A false positive condition would be one where D- 
ICES latches the ICEPro ID mode in a non-iced flight condition. While the impact of an occasional false 
positive is a human factors exercise, the impact of a false negative where the pilot is told the aircraft is 
clean while accreting ice would have serious implications. 

As shown in Eigure D.19 D-ICES allows one false positive indication without filtering. Adding 
sensor noise to selected parameters used in the ICEPro logic, as discussed in Section D.2 causes the Theil 
inequality coefficient to increase, growing larger as the severity of the sensor noise increases. When the 
largest sensor noise was used the Theil inequality coefficient exceeded the ID mode threshold, which 
would cause an IVHM mode switch to ID mode in normal operation. 

Looking at the iced aircraft results without filtering in Eigure D.20, D-ICES fails to indicate iced 
conditions in severe turbulence. This system failure is due to the severity of the turbulence as the aircraft 
stalls and full elevator deflection has been reached by the virtual pilot. The virtual pilot, described in 
Equation (D.l), was unable to maintain airspeed or a level attitude. This is seen in Eigure D.21. Due to the 
lack of aircraft control and severity of the turbulence in the severe turbulence scenarios these conditions 
will not be evaluated. As discussed in Section D.2.2, low virtual pilot gains were used for all conditions 
which worked well without turbulence, but were less than optimal in turbulence. The low virtual pilot 
gains most likely contributed to the aircraft stalling in severe turbulence. Eor Phase I tests, a fixed throttle 
setting was used. However, during Phase II testing, aircraft configuration changes required the throttles to 
be actively controlled. The deactivation of the throttles during Phase I was done to eliminate the excessive 
and frequent throttle movements during flight in turbulence where the control logic was attempting to 
hold velocity in dynamic conditions. 
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Figure D.19. — Unfiltered D-ICES Performance with Clean Aircraft 
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Figure D.20. — Unfiltered D-ICES Performance with Iced Aircraft 
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Figure D.21. — Severe Turbulence Flight Conditions 


D.3.2.1.1 Results of 80 Runs With Fixed Random Seed 

The flight model inversion process in D-ICES is dependent on an input state vector derived from the 
current flight conditions. This state vector includes the following eight elements: 


• Angle of attack 

• Angle of sideslip 

• Roll rate 

• Pitch rate 

• Yaw rate 

• Elevator deflection 

• Aileron deflections 

• Rudder deflection 


Since D-ICES was only used in initial mode switching, it was proposed that filtering these inputs for 
the flight model inversion calculation could he done with a minimal delay on system mode switching. A 
second order low pass filter was chosen with a cut-off frequency of 2.5 Hz. 

The four turbulence scenarios were tested, no turbulence, light turbulence, medium turbulence and 
severe turbulence. As previously explained, the severe turbulence results were omitted from the analysis. 
In each turbulence scenario, sensor noise was added, set to either a nominal condition or off. Einally, a 
fixed random seed, or a reusable series of random numbers, was used to generate turbulence to allow 
direct comparison between the runs. 

Eigure D.22 shows the filtered conditions for the no-ice or clean aircraft with a fixed random noise 
seed. Here the noise influence shown in Eigure D.19 resulting in a false positive has been addressed. In 
addition the effect of atmospheric turbulence on the D-ICES algorithm is minimal. 
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In the iced aircraft scenario, shown in Figure D.23, there were no false negatives with D-ICES. The 
sensor noise effects have been addressed as the scenarios with and without noise are almost over-plots. There 
is a handing effect due to the level of turbulence (i.e., all of the results for a common level of turbulence 
grouped tightly together but at different Theil inequality coefficient values); however, all conditions are well 
above the selected ID Mode switch threshold and result in mode switching at similar times. 
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Figure D.22. — Filtered Clean Aircraft with Fixed Random Seed 
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Figure D.23. — Filtered Iced Aircraft with Fixed Random Seed 
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D.3.2.1.2 


Results of 60 Runs Random Seed 


A total of 60 scenarios were tested (the 20 severe turbulence cases were dropped, as mentioned 
earlier), again at three turbulence levels with one condition without sensor noise and nine additional 
scenarios with sensor noise. In addition, the seed for the random number generation was selected 
randomly creating a unique set of random number for every run. 

In Figure D.24, the clean aircraft conditions continue to exhibit a very tight banding with only three 
turbulence conditions potentially exhibiting a false positive. Further examination revealed that in the 
immediate trim AOA region, small changes in AOA produced very little variation in pitching moment 
due to a specific elevator deflection. If the AOA went to higher values where there was a variation with 
AOA, the Theil inequality coefficient remained well below the threshold value. It was surmised that the 
D-ICES inversion process had some troubles in determining the correct elevator deflection near trim, in 
the presence of light turbulence with these pitch due to elevator characteristics. A real pilot would 
probably not chase the variations in speed and flight path in the same manner as the virtual pilot did in 
these runs, and the results could differ. Looking at only the initial 6 sec after buffer fill {t-6tot- 12) 
provides a snapshot of the typical D-ICES operation. 

Eigure D.25 shows 60 separate runs in D-ICES performance under three turbulence conditions. Under 
light and medium turbulence D-ICES successfully detected the presence of icing conditions without 
degradation or delaying IVHM mode shifting. 
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Figure D.24. — Filtered Clean Aircraft with Random Seed 
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Figure D.25. — Filtered Iced Aircraft with Random Seed 
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D.3.2.2 RT-PID Study Results 

It was found that the application of turhulence and sensor noise on RT-PID had a significant impact 
on performance. As previously mentioned, non-ice test conditions were run, hut in the context of RT-PID 
were considered outside of the scope and omitted from the analysis. Therefore, only the 40 iced runs will 
he reported. Results from the 40 iced runs with fixed turhulence seed are depicted in Figure D.26. The 
error in flow angles was varied from 0.0° to 1.0° in 0. 1° increments. All other errors were fixed using the 
prohe specifications in Table D. 1. 

The results of Figure D.27 illustrate the effect of flow angle error on the coefficient Cm„. As 
expected, increased flow angle error results in increasing error in the estimate and increased error bars 
throughout each turbulence level. The severe turbulence cases are discarded from the analysis since the 
majority of the flight resulted in a stalled condition. 

Next, the MOS message criterion was applied with white Gaussian measurement noise. The following 
results for C^g^, representative of the pitch degrade message, are depicted in Figure D.27 with fixed 
turbulence seed. As the turbulence level increases so does the error in the estimate and the error bounds. 
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Figure D.26. — Effect of flow angle error on Cm„ 
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Figure D.27. — Effect of flow angle error on Cmgg 

D.3.2.3 Task 2 Measures of Success (MOS) 

The MOS message criterion is summarized below for convenience (Appendix E). 

1) Message accuracy 

Message error will be determined by the error bounds and accuracy of its associated stability and 
control coefficient for all ICEPro messages. Coefficients must meet or exceed the non- 
atmospheric turbulence error bound and accuracy in atmospheric turbulence. 

In general, the MOS for message accuracy is not achieved. As seen in Eigure D.27, the estimates 
move away from the truth and error bounds increase with each increase in turbulence level. The same 
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runs that were used to disqualify the MOS message criterion were used to evaluate the ISP MOS criterion 
summarized helow for convenience (Appendix E). 

2) ISP accuracy 

ISP error will he determined hy comparing the true ISP value with the estimated ISP value. ISP is 
a function of five stability and control coefficients: Longitudinal static stability, elevator control 
power, dihedral effect, aileron control power, and normal lift coefficient. The ISP error must meet 
or exceed its non-atmospheric turbulence error bound and accuracy in atmospheric turbulence. 

The following results for ISP, representative of all 40 ice runs with fixed turbulence seed, are 
depicted in Figure D.28. 

Figure D.28 depicts the ISP time history for each simulated flow angle error. The ISP value is initially 
set to zero at the start of each run and ramps up as ice is detected. The D-ICES threshold for determining 
an ice condition is an ISP value of 0.5. This is depicted by dashed red lines in each subplot. Visual 
inspection confirms that the ISP is different in and out of turbulence. In order to satisfy the original MOS 
criteria, the ISP lines in and out of turbulence must be the same. As shown in Figure D.28 this is not the 
case. Additionally the ISP deteriorated greatly with increased flow angle error. In general, the MOS 
criterions for both messages and ISP, as originally defined, were not achieved. A new MOS criterion was 
formulated based on the results depicted in Figure D.27 and Figure D.28. The new criterion takes into 
account the degradation of RT-PID estimates and error bounds in turbulence. Errors in the estimates and 
in the error bounds with turbulence were tolerated as long as the ICEPro messages and AOA brackets are 
equivalently displayed inside and outside of turbulence. The criterion is summarized below. 
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Figure D.28. — ISP for all turbulence levels and flow angle errors (a and (3) 
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1. Compare high and low AOA brackets across turbulence levels. Angle of attack brackets are 
strongly dependent on AOA and flap deflection and weekly dependent on ISP. 

o Equal or conservative bracket location 

- Successful envelope protection 
o Unconservative bracket location 

- Unsuccessful envelope protection 

2. Compare message state across turbulence levels as well as time to latch and time to delatch, 
o Same state annunciation 

- Successful envelope protection 
o Varying state annunciation 

- Unsuccessful envelope protection 

o Latch / delatch within 10 sec of no turbulence results 

- Successful envelope protection 

The following results for message accuracy, representative of AIMMS-20 measurement errors for all 
turbulence levels, are depicted in Figure D.29 using the new MOS criterion. Only the results of the Roll 
degrade message are plotted since the remaining messages did not result in false positives or false 
negatives and were considered successful. In theory, the same messages are expected for all turbulence 
levels since the aircraft is flown about the same reference condition. In Figure D.29, this is represented by 
the average AOA and angle of sideslip during each run. In practice, for zero, light, and moderate 
turbulence the average flow angles are similar. In severe turbulence, the average flow angles are 
significantly different due to wing stall, but do not affect the message accuracy. As stated previously the 
severe turbulence runs are omitted from the analyses and all runs used a fixed turbulence seed for 
comparison purposes. 

In the zero turbulence case, the roll degrade message results in a false positive warning message. This 
is explained by the aileron control power estimate subplot. The estimate of the aileron control power 
quickly diverges from the truth. This difference causes the false positive indication. It should be noted 
that although a false positive condition occurs due to poor estimates the ICEPro logics latches at the 
correct levels. As a side note, this condition was repeated with higher buzzing amplitudes and no false 
positives occurred. In each of the turbulence cases, the additional aileron inputs provided by the virtual 
pilot improved the RT-PID estimates so that no false positives were annunciated. In general the new MOS 
message criterion is considered to be successful. 

Additional testing was conducted to access the new MOS AOA bracket criterion. The following 
results for AOA bracket accuracy, representative of AIMMS-20 measurement errors with conservative 
flow angle errors for all turbulence levels, are depicted in Figure D.30. A conservative flow angle error of 
0.3° versus the stated 0.1° was chosen because experience has proven the difficulty of calibrating up-wash 
and side -wash corrections on air data probes. 

As in the message MOS criterion, the same AOA bracket locations are expected for all turbulence 
levels since the aircraft is flown about the same reference condition. In Figure D.30, these are represented 
by the upper and lower bar delta plots using zero turbulence as the reference. For light and moderate 
turbulence the average deltas are similar. In severe turbulence, the average delta positions are 
significantly different due to wing stall and hence significantly affect the angle of attach bracket location 
and accuracy. The severe turbulence results are omitted from the analysis. 

As shown in Figure D.30, after 25 sec from start-up the ISP oscillates between 0.8 and 1.0. Similarly, 
after 25 sec the AOA brackets oscillate about the zero turbulence AOA brackets. Additional filtering was 
implemented to smooth the AOA brackets in turbulence. The filtered results are shown in Figure D.31. 

The results presented in Figure D.31 are filtered by a low pass fourth order Butterworth filter with a 
cut off frequency of 0. 1 radians per second. Similar plots were generated to determine the effect of flow 
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angle error versus ISP. It was determined that the AOA brackets are strongly dependant on AOA and 
weekly dependant on ISP. Even with perfect instruments the ISP was very similar to the ISP in Figure 
D.31 with 0.3° flow angle error. The trend in Figure D.31 is that the AOA bracket location in turbulence 
oscillates around the zero turbulence case. This agrees with the analysis in Figure D.29 were the average 
AOA and sideslip was approximately the same for all runs except severe turbulence. Closer inspection 
shows that the AOA brackets are conservative and unconservative throughout the runs. Fooking at the 
period after 30 sec, where the ISP has stabilized after start-up, the oscillations are approximately 1° for 
light turbulence and 2° for moderate turbulence. These small changes are not considered to be discernible 
by the pilot. Additionally, since the oscillations are due to the variation in AOA throughout the run and 
not due to the ISP, the new MOS AOA bracket criterion is considered to be successful for this task. 
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Figure D.29. — Roll degrade indication analyses 
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AOA limit indication (0.3° flow vane error) 
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AOA limit indication (0.3°fiow vane error) 





Figure D.31 . — Fiitered AOA bracket indication analyses 
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D.3.2.4 Task 2 Conclusions 
RT-PID 

• The original RT-PID MOS (Appendix E) for message accuracy were not achieved. A new MOS 
criterion was formulated due to results depicted in Figure D.27 and Figure D.28. 

• A new MOS was defined were errors in the estimates and in the error hounds with turbulence are 
tolerated as long as the ICEPro messages and AOA brackets are equivalently displayed inside and 
outside of turbulence. 

• The average flow angles are similar for zero, light, and moderate turbulence. In severe turbulence, 
the average flow angles are significantly different due to wing stall, but do not affect the message 
accuracy. 

• With the exception of the Roll degrade message, there were no false positives or false negatives. 
The roll degrade message was caused during a period of low control activity. In each of the 
turbulence cases, the additional aileron inputs provided by the virtual pilot improved the RT-PID 
estimates so that no false positives were annunciated. 

• In general the new MOS message criterion is considered to be successful. 

D-ICES 

• It was found that the application of turbulence and sensor noise on D-ICES had minimal impact 
on performance without filtering. 

o Only one condition was found that yielded a false positive that would have caused the system 
to enter ID Mode when no icing was present. 

o There was one condition where the elevator deflection Theil inequality coefficient for an iced 
airplane failed to exceed the threshold and two where the value originally exceeded the 
threshold but with time decreased until it was below the threshold. 

• When the state vectors going to the D-ICES inversion routine were filtered to remove higher 
frequency disturbances due to turbulence and noise, D-ICES accurately detected all icing 
conditions in the presence of turbulence and sensor noise and thus operated correctly. 

• Three potential false positive cases were found that showed the elevator Theil Coefficient built up 
over time to exceed the threshold when no ice was present. An examination of the data in this 
region led to supposition that the D-ICES inversion process had some troubles in determining the 
correct elevator deflection with the pitch due to elevator characteristics at those flight conditions. 

Overall it was found that the objective to identify an acceptable error in the wind measurement was 
not required and that the performance of the AIMMS-20 probe was satisfactory for Task 2. 

D.3.3 Task 3 

The purpose of Task 3 was to evaluate four representative phases of flight (Appendix E) with the 
performance results of Task 2. Twenty four test runs were conducted with and without ice, turbulence, 
and wind measurement errors over four phases of flights. Turbulence was tested as calm, light, moderate, 
and severe. Turbulence levels are defined per MIL-F-8785C. All measurement errors were biased in a 
random time history with limits defined by the condition matrix and sensor specifications using the probe 
specifications in Table D.l and the same conservative flow angle errors. 

Whereas Task 2 focused on straight and level flight in the cruise configuration. Task 3 involved 
additional flight phases including, for a reference, a repeat of flaps up straight and level flight, straight 
and level flight flap transition, straight and level flight heading capture, and -3° flight path angle with a 
flap transition. This last phase is representative of the instrument approach task studied with the ICEFTD 
and thirty subject pilots at Embry-Riddle Aeronautical University during the summer of 2009. 

Through initial testing it was found that the Twin Otter in approach configuration (20° flap) was not 
capable of controlled flight in moderate and severe turbulence. During some runs in moderate and all runs 
in severe turbulence, the aircraft entered a wing stall with the virtual pilot. The data from stalled runs was 
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not considered useful. As a result, all moderate and severe turbulence test conditions have been omitted 
from the RT-PID Task 3 analysis. The non-ice test conditions were run, but in the context of RT-PID, 
were considered outside of the scope and omitted from analysis. Additionally, the severe turbulence test 
conditions have been omitted from the D-ICES Task 3 analyses. 

D.3.3.1 D-ICES Study Results 

D.3.3.1.1 Results of 24 Runs Fixed Seed 

The first phase matched the Task 2 conditions. The aircraft was level at 2500 ft, 1 10 kt with the flaps 
retracted. As expected, results in Figure D.32 match Figure D.22 and Figure D.23 in Task 2. 

In the second phase of flight, the aircraft was slowed from an initial velocity of 1 10 to 80 kt and the 
flaps were then extended from 0° to 20°. D-ICFS experienced no false positives during this scenario as 
seen in Figure D.33. The clean aircraft tests experienced little change due to the dynamic flight 
conditions. 

The next phase of flight examined a heading change from 090° to 045°. Here a false negative was 
seen for the iced airplane with no turbulence, as shown in Figure D.34. The value of the Theil inequality 
coefficient for elevator deflection is a measure of how well the actual elevator time history matches the 
predicted time history for an uniced airplane. For an uniced airplane, the elevator deflection time history 
should be virtually identical to the D-ICES predicted time history. This will produce very low values of 
the dell Theil inequality coefficient. A sample of the elevator and corresponding Theil inequality 
coefficients for the uniced airplane with light turbulence is shown in Figure D.35. In the lower plot, 
INV_PitchCmd is the predicted elevator deflection, while 8g is the actual deflection. 


Task 2 

Level flight, 2500 ft, llOkt, 5^:0° 
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Task 2 

Level flight, 1 1 0 kt ^ 80 kt, 5^: 0° ^ 20° 



Task 2 

Level flight, left turn 090° ^ 045° 80 kt, 5^: 20° 



For an iced airplane, there should he some differences between the expected (uniced) elevator 
deflection and the actual deflection. The closer the iced and uniced C,„^^ values are, the less this will he. 
In Figure D.36, one can see that the actual and predicted elevator deflection were starting to separate 
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resulting in the larger Theil inequality coefficient values. However, because the iced Cmg^ was not very 
different from the uniced values at this flight condition, the differences without turbulence were not 
enough to exceed the threshold. With the addition of turbulence, the elevator deflection separation 
became greater and resulted in Theil inequality coefficient values that exceeded the threshold. This is 
probably because the turbulence oscillations caused changes in alphas where there was greater difference 
between the iced and uniced Cm^g’s- 

In the fourth phase of flight, the aircraft was flown down a 3° glide slope from 3000 ft at 75 kt. As the 
descent was established, the airspeed was reduced from 75 to 70 kt and flaps were then extended to 30°. 
While there were no false indications. Figure D.37 shows that the D-ICES results remained close to the 
singular condition seen in the level turning flight case condition. However, the Theil inequality coefficient 
values are still sufficient to exceed the threshold for transitioning to ID mode for all of the iced 
conditions. 
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Figure D.35. — Uniced Elevator Deflection Theil inequality Coefficient Analysis 
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Figure D.36. — Iced Elevator Deflection Theil inequality Coefficient Analysis 
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Task 2 

3° descent from 3000 ft, 75 kt ->• 70 kt, 5?: 20° ^ 30° 




D.3.3.1.2 Results of 24 Runs Random Seed 

Task 3 was rerun using a random seed. Figure D.38, summarizes the results with all four scenarios 
plotted. Due to occasional low speed stall issues with medium turbulence with the random seed, these 
conditions were excluded. As expected, the condition that caused the iced aircraft with no turbulence to 
exhibit a false positive during the heading capture scenario still produced similar results here. Also, with 
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the random seed, the same scenario light turhulence condition was delayed hy 10 sec before latching into 
ID Mode. 

D.3.3.2 RT-PID Study Results 

In general, the original MOS criterion for both messages and ISP are not achieved in Task 3 and 
results are similar to Figure D.27 and Figure D.28 in Task 2. As in Task 2 the new MOS criterion based 
on AOA brackets and messaging was applied to each flight phase. The results for this task are broken up 
by respective flight phase. 

D.3.3.2.1 Phase 1 — Flaps Up Straight and Level Flight 

As expected from Task 2 results. Phase 1 runs had no pitch degrade, climb limit, or flap limit 
messages for zero, light, and moderate turbulence levels. 

As shown in Figure D.39, the light turbulence roll degrade message appears before the zero 
turbulence roll degrade message. Since this is within the 10 sec defined in the MOS this is considered 
successful. Like Figure D.29 in Task 2, Figure D.39 shows false positives for light and no turbulence 
cases. This condition was repeated with higher buzzing amplitudes and no false positives occurred. The 
MOS criterion for AOA bracket was applied to Phase 1 with similar results as shown in Task 2 Figure 
D.31 and considered to be successful. 
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Figure D.39. — Phase 1 Roll degrade indication analyses 
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D.3.3.2.2 Phase 2 — Straight and Level Flight Flap Transition 

Phase 2 runs had no pitch degrade, yaw degrade, climh limit, or flap limit messages for zero, light, 
and moderate turhulence levels. As shown in Figure D.40 the roll degrade message performance in 
turbulence was better than outside of turbulence. In light and moderate turbulence the system did not 
annunciate any caution or warning messages. The increase in virtual pilot activity in turbulence during the 
flap transition provided improved RT-PID performance to that shown in Figure D.40. In Figure D.40 the 
bottom plot shows poor RT-PID performance in zero turbulence with respect to the truth values due to 
lack of information content. 

As in Task 2, the same AOA bracket locations are expected for all turbulence levels since the aircraft 
is flown about the same reference condition. In Figure D.41, these are represented by the upper and lower 
bar delta plots using calm atmosphere as a reference. After approximately 35 sec the flap transition from 
zero to 20° flap is complete. Incidentally it takes approximately the same amount of time for the ISP to 
stabilize from start-up as shown at the bottom of Figure D.41. It should be noted that the AOA brackets 
exhibit similar behavior to that found in Task 2 from 35 to 60 sec and therefore considered to be 
successful. There are no pitch degrade messages generated during this time because the current ICEPro 
logic for pitch degrade messaging is based on the elevator effectiveness, C,„gg, which is well within the 
thresholds such that no messages are generated. Data after 60 sec should be discarded because the virtual 
pilot is unable to adapt near the tail stall boundary and enters into a pilot coupling event between repeated 
incipient tail stalls and wing stalls causing the large oscillations shown. Along these lines it should be 
noted that the virtual pilot gains were set as low as practical while obtain reasonable performance without 
excessive control activity. A human pilot who is familiar with the aircraft and its flying characteristics 
may be better suited to control these situations. It is recommended that additional piloted runs be 
conducted to verify this conclusion by the virtual pilot. 
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Figure D.40. — Phase 2 Roll degrade indication analyses 
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Figure D.41 . — Phase 2 AOA bracket analyses 


D.3.3.2.3 Phase 3 — Straight and Level Flight Heading Capture 

Phase 3 runs had no pitch degrade, climh limit, or flap limit messages for zero, light and moderate 
turbulence levels. As shown in Figure D.42 the roll degrade message performance in turbulence was 
better than out of turbulence. In light turbulence the system annunciates a caution message and in 
moderate turbulence the system did not annunciate any caution or warning messages. The increase in 
virtual pilot activity during the heading capture due to turbulence provided improved RT-PID 
performance. In Figure D.42 the bottom plot shows poor RT-PID performance in zero turbulence with 
respect to the truth values due to lack of information content. 

Figure D.43 shows a typical run for Phase 3, heading capture. Ice did not greatly affect the rudder 
effectiveness for this airplane, so it was anticipated that there would be few, if any yaw degrade messages 
as is shown to be the case in Figure D.43 no messages were generated for any of the heading capture 
conditions investigated. 
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Figure D.42. — Phase 3 Roll degrade indication analyses 
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The MOS criterion for AOA bracket was applied to Phase 3 with similar results as shown in Task 2, 
Figure D.31 and Task 3, Figure D.41 and considered to be successful. Similar to Task 3 Figure D.41 the 
last 20 sec should be discarded because the virtual pilot is unable to adapt near the tail stall boundary and 
enters into a pilot coupling event between repeated incipient tail stalls and wing stalls causing large 
oscillations. 

D.3.3.2.4 Phase 4 — Descending Flight Flap Transition — 3° Glide Slope 

Phase 4 runs had no pitch degrade or yaw degrade messages for zero, light or moderate turbulence 
levels. As shown in Figure D.44 the roll degrade message performance in turbulence was better that out 
of turbulence. The increase in virtual pilot activity during the heading capture due to turbulence provided 
improved RT-PID performance. In Figure D.44 the bottom plot shows poor RT-PID performance in zero 
turbulence with respect to the truth values due to lack of information content. 

As shown in Figure D.44 the zero turbulence roll degrade caution and warning messages annunciate 
before the light turbulence messages. This is considered to be conservative and successful based on MOS 
criterion. The toggle in the warning message in moderate turbulence is discarded because as before the 
virtual pilot is unable to adapt near the tail stall boundary and enters into a pilot coupling event between 
repeated incipient tail stalls and wing stalls. Because the elevator effectiveness during this maneuver stays 
within the thresholds for caution and warning messages, no pitch degrade messages are generated. 
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As shown in Figure D.45 the zero turbulence climh limit warning message annunciated at the same 
time as in light turhulence. This is considered to he successful based on the MOS criterion. In moderate 
turbulence the warning message came on for approximately 7 sec. The ICEPro message logic requires 
messages to persist for 10 sec before they can unlatch, however, when a stall is detected by ICEPro, it 
declutters the PED by removing all other messaging for 20 sec to allow the pilot to regain normal flight. 
After that time, ICEPro returns to normal messaging. In the situation shown in Eigure D.45, a stall 
condition occurred which terminated the messaging. 

As shown in Eigure D.46 the zero turbulence flap limit warning message annunciates at the same time 
as in light turbulence. This is considered to be successful based on the MOS criterion. As in the climb 
limit message, in the moderate turbulence case the warning message came on for approximately 7 sec. 

The ICEPro message logic requires messages to persist for 10 sec before they can unlatch. The stall 
condition resulted in decluttering the PED and removed the message after 7 sec, as discussed above. 

The MOS criterion for AOA bracket was applied to Phase 4 and shown in Eigure D.47. After 
approximately 30 sec the flap transition from 20 to 30° flap is complete. Incidentally it takes 
approximately the same amount of time for the ISP to stabilize from start-up as show at the bottom of 
Eigure D.47. It is believed that the increased oscillations in Phase 4 AOA brackets over previous phases is 
because of the narrow envelope between wing stall and tail stall at flaps 30°. The inability to adapt by the 
virtual pilot in this reduced stability regime causes larger oscillations. Applying the MOS criterion would 
be unsuccessful in this phase since a pilot is able to discern the unconservative variation in the AOA 
brackets. It is recommended that additional piloted runs be conducted to verify this conclusion by the 
virtual pilot. 
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Figure D.45. — Phase 4 Climb Limit indication analyses 
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Single Engine ROC Caution 
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Figure D.46. — Phase 4 Flap Limit indication analyses 
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Figure D.47. — Phase 4 AOA bracket analyses 
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D.3.3.3 Task 3 Conclusions 
D-ICES 

• The first scenario was a repeat of Task 2 conditions and the results were similar. 

• For the level flight flap extension scenario, the aircraft was slowed from an initial velocity of 1 10 
to 80 kt and the flaps were then extended from 0° to 20°. D-ICES experienced no false positives 
during this scenario as seen in Figure D.33 

• The next scenario examined was a heading change from 090° to 045°. Here a false negative was 
seen for the iced airplane with no turbulence hut correctly predicted positives for the low and mid 
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turbulence levels. The false positive was believed to be due to a singular condition where there 
was little difference between the iced and clean airplane’s control effectiveness at the chosen 
flight condition. 

• For the 3° glide slope descent scenario there were no false indications, even though the results 
remained close to a singular condition seen in the level turning flight case. However, the Theil 
inequality coefficient values are still sufficient to exceed the threshold for transitioning to ID 
mode for all of the iced conditions. 

• In most cases, D-ICES performed as designed in the presence of turbulence and noise. However, 
the exceptions noted above, especially the false negatives, do indicate that some further 
development of the system is warranted. One possible area to examine would be utilizing other 
terms related to the Theil inequality coefficient, namely the bias, variance and co-variance terms 
in some way to develop a more robust detection scheme. 

RT-PID 

• Phase 1 runs had no pitch degrade, climb limit, or flap limit messages for zero, light, and 
moderate turbulence levels. The MOS criterion for AO A bracket was applied to Phase 1 and 
considered to be successful. 

• Phase 2 runs had no pitch degrade, yaw degrade, climb limit, or flap limit messages for zero, 
light, and moderate turbulence levels. The MOS criterion for AOA bracket was applied to Phase 
2 and therefore considered to be successful. 

• Phase 3 runs had no pitch degrade, climb limit, or flap limit messages for zero, light, and 
moderate turbulence levels. The MOS criterion for AOA bracket was applied to Phase 3 and 
considered to be successful. 

• Phase 4 runs had no pitch degrade or yaw degrade messages for zero, light, and moderate 
turbulence levels. As shown in Figure D.46 the calm atmosphere flap limit warning message 
annunciates at the same time as in light turbulence. This is considered to be successful based on 
the MOS criterion. The MOS criterion for AOA bracket was applied to Phase 4: Applying the 
MOS criterion would be unsuccessful in this phase since a pilot is able to discern the 
unconservative variation in the AOA brackets. 

• It is recommended that additional piloted runs be conducted to verify this conclusion by the 
virtual pilot. 

In general, it was found that the performance of the AIMMS-20 probe was satisfactory for Task 3. 

D.4 Conclusions and Recommendations 

Summaries are provided of the results of each task. For Task 2 and Task 3 the summaries are divided 
into D-ICES and RT-PID. 

D.4.1 Summary of Results 
Task 1 

Several practical issues related to real-time parameter estimation for a linear longitudinal dynamics 
model in atmospheric turbulence using indirect turbulence measurements were examined and discussed. 
The frequency response of the airflow vanes, wing response to atmospheric turbulence, and the structural 
response of the air data boom were identified as sources of colored noise. The frequency response of the 
airflow vanes for a wide variety of combinations of natural frequency and damping ratio did not adversely 
impact parameter estimation results in turbulence, except for Czq and Cz^e- This was attributed to noisy 
explanatory variables in the atmospheric turbulence implementation and relatively low aerodynamic 
contributions as compared to Cza- Airflow vane dynamics do not substantially contribute to the scatter in 
the estimates, because all of the added dynamics are outside the range of frequencies analyzed. The 
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structural modes of the boom lead to similar conclusions, because the natural frequencies of the boom 
structural modes are outside the range of frequencies used for dynamic modeling. A major source of 
colored noise was identified as the frequency-dependent upwash and time delay induced by the wing- 
bound vortex system and the longitudinal separation between the AOA measurement and the wing. It was 
also surmised that the frequency-dependent sidewash was a major source of colored noise. The 
frequency-dependent upwash and time delay appear to be significant contributors to biasing and increased 
scatter and uncertainty for parameter estimates in atmospheric turbulence. 

Practical issues were examined using data from a Twin Otter DHC-6 longitudinal linear simulation, 
with realistic noise sequences added to the computed aircraft responses. This allowed a clear view of the 
effect of each source of colored noise to the modeling problem, because the true values of the model 
parameters were known. This approach was used initially to show that real-time parameter estimation can 
be done accurately in all atmospheric turbulence conditions if the AOA measurement is accurate. Flight 
test data from the GRC DHC-6 Twin Otter aircraft was used to validate the effect of the identified colored 
noise sources. 

Based on these findings, several practical recommendations for flight testing in atmospheric 
turbulence are suggested, as well as areas for future study. First, mount the air data boom at the nose of 
the aircraft to minimize the effect of the wing-bound and wing-tip vortices. Second, if the air data boom is 
mounted on the nose of the aircraft, an upwash and time delay calibration as a function of frequency is 
required. 

Task 2 

D-ICES 

• It was found that the application of turbulence and sensor noise on D-ICES generally had a 
minimal impact on performance without filtering. 

o Only one condition was found that yielded a false positive that would have caused the system 
to enter ID Mode when no icing was present. 

o There was one condition where the elevator deflection Theil inequality coefficient for an iced 
airplane failed to exceed the threshold and two where the value originally exceeded the 
threshold but with time decreased until it was below the threshold. 

• When the state vectors going to the D-ICES inversion routine were filtered to remove higher 
frequency disturbances due to turbulence and noise, D-ICES accurately detected all icing 
conditions in the presence of turbulence and sensor noise and thus operated correctly. 

• Three potential false positive cases were found that showed the elevator Theil Coefficient built up 
over time to exceed the threshold when no ice was present. An examination of the data in this 
region led to supposition that the D-ICES inversion process had some troubles in determining the 
correct elevator deflection with the pitch due to elevator characteristics at those flight conditions. 


RT-PID 

• In general, the (Old) RT-PID MOS for message accuracy was not achieved. A new MOS criterion 
was formulated due to results depicted in Eigure D.27 and Eigure D.28. 

• Errors in the estimates and in the error bounds with turbulence are tolerated as long as the ICEPro 
messages and AOA brackets are equivalently displayed inside and outside of turbulence (New 
MOS). 

• In practice, for zero, light, and moderate turbulence the average flow angles are similar. In severe 
turbulence, the average flow angles are significantly different due to wing stall, but do not affect 
the message accuracy. 

• With the exception of the Roll degrade message, there were no false positives or false negatives. 

• In each of the turbulence cases, the additional aileron inputs provided by the virtual pilot 
improved the RT-PID estimates so that no false positives were annunciated. 
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• In general the new MOS message criterion is considered to be successful. 

Overall it was found that the objective to identify an acceptable error in the wind measurement was 
not required and that the performance of the AIMMS-20 probe was satisfactory for Task 2. 

Tasks 

D-ICES 

• For the level flight flap extension scenario, the aircraft was slowed from an initial velocity of 1 10 
to 80 kt and the flaps were then extended from 0° to 20°. D-ICES experienced no false positives 
during this scenario as seen in Figure D.33. 

• The next scenario examined was a heading change from 090° to 045°. Here a false negative was 
seen for the iced airplane with no turbulence but correctly predicted positives for the low and mid 
turbulence levels. The false positive was believed to be due to a singular condition where there 
was little difference between the iced and clean airplane’s control effectiveness at the chosen 
flight condition. 

• For the 3° glide slope descent scenario there were no false indications, even though the results 
remained close to a singular condition seen in the level turning flight case. However, the Theil 
inequality coefficient values are still sufficient to exceed the threshold for transitioning to ID 
mode for all of the iced conditions. 


RT-PID 

• Phase 1 runs had no pitch degrade, climb limit, or flap limit messages for zero, light, and 
moderate turbulence levels. The MOS criterion for AOA bracket was applied to Phase 1 and 
considered to be successful. 

• Phase 2 runs had no pitch degrade, yaw degrade, climb limit, or flap limit messages for zero, 
light, and moderate turbulence levels. The MOS criterion for AOA bracket was applied to Phase 
2 and therefore considered to be successful. 

• Phase 3 runs had no pitch degrade, climb limit, or flap limit messages for zero, light, and 
moderate turbulence levels. The MOS criterion for AOA bracket was applied to Phase 3 and 
considered to be successful. 

• Phase 4 runs had no pitch degrade or yaw degrade messages for zero, light or moderate 
turbulence levels. As shown in Figure D.46 the calm atmosphere flap limit warning message 
annunciates at the same time as in light turbulence. This is considered to be successful based on 
the MOS criterion. The MOS criterion for AOA bracket was applied to Phase 4: Applying the 
MOS criterion would be unsuccessful in this phase since a pilot is able to discern the 
unconservative variation in the AOA brackets. 

• It is recommended that additional piloted runs be conducted to verify this conclusion by the 
virtual pilot. 

In general, it was found that the performance of the AIMMS-20 probe was satisfactory for Tasks 2 and 

3. Acceptable error is therefore defined by current specification of AIMMS-20 probe. As long as booms are 

not used and the calibration for wing response to turbulence has been taken care of the system will work. 

D.4.2 Additional Questions Raised by Research 

1. What is the calibration accuracy required of the wing response to turbulence for RT-PID estimates to 

be within +10 percent? 

2. What is the boom stiffness required for RT-PID estimates to be within +10 percent 

3. How can the other terms related to the Theil inequality coefficient, namely the bias, variance and co- 

variance terms be used to develop a more robust detection scheme. 
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D.4.3 Recommended Future Studies 


1. Execute Task 4. This task evaluates the effectiveness of manual pilot inputs alone for making high 
confidence vehicle state estimates when performing typical maneuvering flight tasks under varying 
levels of atmospheric turbulence. 

2. Flight test to validate RT-PID findings based on colored AOA regressor findings. 

3. Flight test to validate ICEPro. 
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E.l Abstract 


The effects of atmospheric turbulence on the RT-PID and D-ICES algorithms are studied hy 
modeling representative atmospheric turhulence. Atmospheric turbulence is measured directly as an input 
to both methods. The performance of these algorithms is evaluated using reasonable assumptions for 
measurements from currently-available turbulence sensors in operational flight. Methods for improving 
the performance of these algorithms in these conditions are implemented and studied. 

E.2 Introduction 
E.2.1 Background 

A team comprised of personnel from The University of Tennessee Space Institute (UTSI), Tullahoma, 
Tennessee, Bihrle Applied Research Inc (BAR), Hampton, Virginia, NASA Glenn Research Center 
(GRC), Cleveland, Ohio, and NASA Langley Research Center (LaRC), Hampton, Virginia, has 
successfully developed a prototype Ice Contamination Envelope Protection System (ICEPro). The ICEPro 
system facilitates flight envelope protection by making continuous real time vehicle state assessments, 
which are synthesized into flyable pilot cueing along with visual and aural alerts during in-flight icing 
conditions. Detection of degraded aircraft stability and control and performance due to icing is carried out 
by a dynamic inversion control evaluation system (D-ICES) that compares expected aircraft behavior 
from a-priori knowledge base with current measures of those behaviors. When differences reach defined 
thresholds, real time parameter identification (RT-PID) methods are invoked to estimate current state 
parameters, which continuously support pilot cueing and alerts. The development effort included 
simulation-based design, testing, and verification. A pilot in-the-loop study was conducted to gather pilot 
performance data and opinions of the utility of ICEPro during simulated icing encounters. Results of the 
study indicated that the system performed as expected and pilot performance benefited from the envelope 
protection cues. However, the simulation could not duplicate all of the real-world conditions that could 
have first-order effects on system performance, such as those due to real atmospheric turbulence. 
Therefore, in order to assess system performance under these conditions and minimize the technical risk 
of an eventual flight-test validation program, the effects of atmospheric turbulence on ICEPro are studied. 

E.2.2 Participating Organizations 

The following organizations will provide simulator flight test support and facilities. 

UTSI: Is one of the primary organizations and facilities for simulator flight testing and algorithm 

development of the ICEPro system. The basis for the simulator flight tests will be a COTS 
desktop simulator running BAR’s D-six software and representative flight controls, visuals, and 
virtual aircraft instruments. 

BAR: Is one of the primary organizations for algorithm and software development of the ICEPro 

system. The basis for algorithm and software development will be a COTS desktop simulator 
running BAR’s D-Six software and representative flight controls, visuals, and virtual aircraft 
instruments. 

GRC: Will provide technical monitoring and serve as the NASA focal point for this grant. 

LaRC: Will provide technical assistant with real-time parameter identification algorithms. 

E.2.3 Overview of Test Concepts 
Simulator Flight Test 

The ICEPro systems software will be tested in Birhle’s D-six software environment. Software will be 
tested both at UTSI and BAR. In support of these tests, BAR will provide the following: 

1. Update the ICEPro software so that desktop simulations at BAR and UTSI match the ICEETD. 
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2. Update the ICEPro system logic diagram. 

3. Synchronize system software between UTSI and BAR as software changes are made. 

4. Author release notes documents with each software revision and update the system logic diagram 
were appropriate. 

E.2.4 Outline of Document 

The remainder of this document is divided into two parts. Section E.3 contains the simulator flight 
test plan and Section E.4 contains the simulator flight test tasks. 

E.3 Simulator Flight Tests 
E.3.1 Test Objectives 

The primary ohjectives of the simulator flight tests are to assess the performance of RT-PID and D- 
ICES algorithms in the presence of atmospheric turbulence and determine methods for improving 
algorithm performance in these conditions. The simulations will be based on events that cannot be 
repeatedly duplicated during flight testing and benefit from a controlled environment. 

The secondary objective is to access the performance of RT-PID and D-ICES algorithms in the 
absence of an onboard excitation system (OBES) with and without atmospheric turbulence. 

E.3.2 Test Guidelines 

E.3.2.1 Primary Objectives 

E.3.2.1.1 Performance Testing in Turbulence 

The performance of the RT-PID and D-ICES algorithms will be assessed by flying a low gain and a 
high gain pilot-in-the-loop task in the presence of atmospheric turbulence. The low gain task is straight 
and level unaccelerated flight with the flaps retracted. The high gain task is the final segment of an 
instrument approach. This represents a descent from the final approach fix to the inner marker and 
requires tight control of the glide slope and localizer. Both tasks will be performed under Visual 
Meteorological Conditions (VMC) conditions. Prior to these tasks, a UTSI pilot will train to ensure 
proficiency in basic instrument flying skills and can perform a precision approach within Airline 
Transport Pilot (ATP) standards. 

E.3.2.1.2 Development of Algorithms for Atmospheric Turbulence 

To develop RT-PID and D-ICES algorithms in the presence of atmospheric turbulence this study will 
utilize the previous pilot in-the-loop test scenario. A copy of this scenario is found in Eigure E. 1 . 
However, to coordinate testing and algorithm and software development between UTSI and BAR, the test 
scenario will be broken up into a series of segments or phases of flight. These will consist of straight and 
level unaccelerated flight, flap transition, heading capture, descent, and glide slope and localizer tracking. 
The use of autopilots will provide a standardized pilot performance model for batch testing at UTSI and 
BAR. 

E.3.2.2 Secondary Objectives 

E.3.2.2.1 Preliminary Eeasibility Study Without OBES 

Einally, the updated RT-PID and D-ICES algorithms will be assessed in the in the presence of 
atmospheric turbulence, with a pilot-in-the-loop, with and without an on board excitation system. The 
algorithms will be assessed by the same pilot-in-the-loop low gain and high gain tasks as in the initial 
task. Additionally, a UTSI pilot will be under the same training and flying standards as previously 
mentioned. 
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E.3.3 Performance Parameters to be Measured 

All flight data will be recorded at 50 Hz. The following parameters will be recorded and calculated. 

1. OBES frequency and duration 

2. ISP value 


Error in ISF 

' parameters 

a. 

Cnia 

Longitudinal Static Stability 

b. 


Elevator Control Power 

c. 

C/(3 

Dihedral Effect 

d. 

^l5a 

Aileron Control Power 

e. 

CNa 

Normal Lift Coefficient 


4. Error in Message parameters 

5. Glide slope and localizer performance 

6. Airspeed performance 

7. Number of AOA exceedances 

a. Stick shaker and stick pusher firings 

b. STALL message postings 

c. Tail stall events 

E.3.4 Measures of Success (MOS) 

To ensure that test objectives are attained, MOS’s have been formulated. A MOS is defined as a 
comparative requirement which must be satisfied by the results of testing. The overall measure of success 
for the primary and secondary objectives will be that the error bounds and accuracy in RT-PID and Theil 
inequality coefficients in D-ICES algorithms meets or exceeds its non-atmospheric turbulence 
performance in atmospheric turbulence. This success criterion focuses on the performance of these two 
algorithms and not on ICEPro’s utility to mitigate a potentially hazardous icing encounter or on the pilot’s 
performance during such encounter. It assumes that if the RT-PID and D-ICES algorithms perform 
equally in and out of atmospheric turbulence the ICEPro system will perform equally. The following 
MOS are defined for the primary objective. 

RT-PID Algorithm 

5. Message accuracy 

Message error will be determined by the error bounds and accuracy of its associated stability and 
control coefficient. Eor example, the red pitch degrade logic is a function of elevator control power 

Cmie — 0.25CmSc_ .no ice 

The elevator control power must meet or exceed its non-atmospheric turbulence error bound and 
accuracy in atmospheric turbulence. This is repeated for all ICEPro messages. 

6. ISP accuracy 

ISP error will be determined by comparing the true ISP value with the estimated ISP value. ISP is a 
function of five stability and control coefficients: Longitudinal static stability, elevator control power, 
dihedral effect, aileron control power, and normal lift coefficient. The true ISP value is known since 
the five true stability and control parameters are known to the simulation. The ISP error must meet or 
exceed its non-atmospheric turbulence error bound and accuracy in atmospheric turbulence. 
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D-ICES Algorithm 
2. Control deflection accuracy 

Control deflection error is determined by comparing the estimated deflection Theil inequality 
coefficient with the baseline no turbulence case. The control deflection error must meet or exceed its 
non-atmospheric turbulence error in atmospheric turbulence. 

E.3.5 Organizational Responsibilities 

The organizational responsibilities for each task are stated in Section E.4. Through each task the 
responsible party’s name is highlighted by parenthesis. 

E.4 Simulator Flight Test Tasks 
E.4.1 Task 1 (UTSI) 

Determine whether the scatter in the turbulence estimates arise from, 1) colored noise in the 
regressor(s), and if so what is the source of the colored noise and in which regressor(s), 2) insufficient 
model structure, and if so how should the model structure be adjusted or, 3) some other means. 

E.4.2 Primary Objective — Performance Testing in Turbulence 

Accomplished in Task 2 by baseline turbulence 

E.4.3 Primary Objective — Development of Algorithms for Atmospheric Turbulence 
E.4.3.1 Task 2 (UTSI and BAR) 

Modify, test, and validate the use of RT-PID and D-ICES algorithms with atmospheric turbulence as 
an input for various levels of atmospheric turbulence. Verify MOS during each phase of flight for varying 
atmospheric turbulence levels and icing conditions. UTSI and BAR will use the full 6-DOE nonlinear 
twin otter database. 

Autopilot Task (Batch Testing) 

1. Isolate the RT-PID problem from D-ICES and executive logic (UTSI) 

a. Elaps up straight and level flight for 30 sec at 1 10 KIAS and 2500 ft 

(1) Perfect Turbulence measurement — Optimal Inputs 

(a) Turbulence levels 

i. Baseline (no turbulence) 

ii. Light 

iii. Moderate 

iv. Severe 

(b) Ice 0 and 2 

(2) Degraded Turbulence measurement — Optimal Inputs 

(a) Turbulence levels 

i. Baseline (no turbulence) 

ii. Light 

iii. Moderate 

iv. Severe 

(b) Ice 0 and 2 

(c) Vary turbulence measurement error 0.3 to 1.5 kt in increments of 0.3 kt 

2. Isolate the D-ICES problem from RT-PID and executive logic (BAR) 
a. Repeat steps outlined in 1 
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Software Requirements (Task 2 plus) 

1. D-six Light, Moderate, and Severe Turbulence Models (BAR) 

2. D-six level flight autopilot with OBES hook outside of the autopilot loop (BAR) 

3. D-six software switch providing isolation of D-ICES or RT-PID algorithms (BAR) 

4. Example script file that calls D-six within MATLAB, trims linear model to 1 10 KIAS, isolates an 
algorithm, selects a turbulence model, selects and icing state, runs the model for a given time, and 
saves one example parameter (BAR) 

5. Batch script file for RT-PID and D-ICES testing (UTSI and BAR respectively) 

6. Update RT-PID and D-ICES algorithms as required (UTSI and BAR respectively) 

E.4.3.2 Task 3 (UTSI and BAR) 

Modify, test, and validate the use of RT-PID and D-ICES algorithms with atmospheric turbulence as 
an input. Select an acceptable atmospheric turbulence measurement error based on findings from Task 2 
and commercially available atmospheric turbulence sensors. Verify MOS during each phase of flight for 
varying atmospheric turbulence levels and icing conditions. UTSI and BAR will use the full 6-DOF 
nonlinear twin otter database. 

Autopilot Task (Batch Testing) 

Phases of flight — Autopilot with Optimal Inputs 

1. Flaps up straight and level flight at 1 10 KIAS and 2500 ft — 30 sec duration 

2. Flaps up straight and level flight at 1 10 KIAS and 2500 ft followed by a constant altitude flap 
transition (0° to 20°) to 80 KIAS - 1 min duration 

3. Heading capture at 80 KIAS, 2500 ft, and flaps 20° from 090° to 360° - 30 sec duration 

4. Descent from 3000 ft at 75 KIAS and flaps 20° followed by a flap transition (30°) to 70 KIAS with 
localizer and glide slope tracking - 2 min duration 

Ice Condition 

1. IceO 

2. Ice 2 in zero time 
Turbulence Levels 

1. Baseline (no turbulence) 

2. Light 

3. Moderate 

4. Severe 

Software Requirements (Task 1 plus) 

1 . D-six autopilots for Phases 2 to 4 (BAR) 

2. Update RT-PID and D-ICES algorithms as required (UTSI and BAR respectively) 

E.4.4 Secondary Objective — Preliminary Eeasibility Study without OBES 
E.4.4.1 Task 4 (UTSI and RR) 

Research Objectives 

1 . Evaluate the effectiveness of manual pilot inputs alone for making high confidence vehicle state 
estimates when performing typical maneuvering flight tasks under varying levels of atmospheric 
turbulence. 

a. Determine the portion or portions of the maneuvers, which provide frequency content at the 
natural frequency of the short period, dutch-roll, and roll modes and thereby support the 
achievement of high confidence parameter estimates from RT-PID: 

(1) Without use of the OBES. 
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(2) With OBES. 

(3) Optionally, acquire pilot workload assessment for (1) and (2) 

Assumptions 

1. Accomplishment of this objective assumes that the goal of system performance as stated in the 
primary ohjectives has been met. 

Test Design 

1 . This test will be designed and executed to acquire data that will enable the researcher to compare the 
differences between the effect of manual only pilot control inputs and pilot inputs that are augmented 
by the operation of the OBES on RT-PID vehicle state estimates. This comparison will be made for 
four atmospheric turbulence conditions, which were previously defined under Task 1. The required 
data will come from maneuver time histories of pilot control inputs with and without OBES 
“buzzing” and from RT-PID derived confidence intervals of key stability and control coefficients in 
each of the aircraft’s three rotational axes. The a priori truth versus the current estimate for each 
stability and control coefficient in the ISP formulation will be used to determine the accuracy in the 
estimate and will be reflected in the magnitude error bars for each coefficient of interest. This will 
form the basis for pilot alerting when the state estimate needs to be improved. A power spectral 
density analysis of control activity with and without OBES, for each part of the maneuver where a 
poor vehicle state estimate exists will be used to determine the amount of energy in the inputs at the 
desired frequency. Coherence plots will then be developed will facilitate the comparison between the 
non-OBES and OBES control input condition. 

Method of Test 

1. Task 4 will be divided into two subtasks. 

a. The first subtask will be an initial evaluation of normal manual pilot inputs for achieving vehicle 
state estimates while performing basic flight maneuvers. These maneuvers will include steady 
level flight, climbs and descents, changes in airspeed, turns to headings, wing flap 
extensions/retractions, and power transitions. These flight maneuvers are described in the 
following section, and will be performed with and without OBES operation. This subtask will be 
performed in smooth atmospheric conditions, and later, under various levels of atmospheric 
turbulence. The goal of this testing is to acquire data that establishes the relationship between 
pilot inputs alone, and pilot inputs with OBES when performing basic aircraft maneuvers, and 
their effect on achieving the required level of confidence in vehicle state estimation. The results 
of this initial testing will be used to verify system functionality, ensure readiness for more 
extensive data collection as described in the second subtask below, and to verify data collection, 
processing, and analysis methods In addition, this testing will identify and address any anomalies 
found with pilot cueing and messaging such nuisance alerts and toggling. 

b. The second subtask will be an extensive series data collection tests flown by a test pilot to 
evaluate manual pilot control input gain on vehicle state estimates under various conditions of 
atmospheric turbulence. This will involve three repeats of each maneuver after the pilot has 
practiced and reached a consistent level of performance. These tests will be performed both with 
and without OBES operation. Two means of varying pilot control activity and gain will be 
employed during this subtask. The first is by increasing desired maneuver performance 
requirements, which involves specifying high and low error tolerances during each set of repeat 
maneuvers. The large tolerance will result in lower gain and control activity, and the small 
tolerances should result in higher gain and control activity. A second means of affecting pilot gain 
and control activity will be to simply alert the pilot via TBD methods (voice or possibly 
messaging) that more aggressive inputs are required in any or all of the pitch, roll, and yaw axes 
until the alerting requirements no longer exist. It is also expected that some maneuvers may not 
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provide sufficient vehicle excitation in all the aircraft axes, e.g., level climhs and descents may 
not provide lateral directional excitation, and it may he necessary to combine the two methods to 
obtain a satisfactory state estimate. In this case, the pilot may be alerted to make inputs in the 
affected axes if a full state estimate is desired for that test point. As a final evaluation of manual 
pilot inputs, an operationally representative flight profile consisting of an instrument approach 
procedure as shown in Figure E.l will be performed under each of the turbulence conditions 
defined in this plan. In addition to evaluating the effectiveness of manual pilot inputs on vehicle 
state estimation, the data derived from this subset of testing will also assess the effect of manual 
inputs and OBES operation on pilot workload, 
c. Pilot tasks will be performed under simulated IMC conditions. 


Data Requirements 


1 . 


Test data will be sampled at a minimum of 10 Hz. The sampled parameters include time histories of 


the following: 
a. Pressure altitude 

Hp 

b. 

Airspeed 

VIAS 

c. 

Angle of attack (AOA) 

a 

d. 

Angle of sideslip (AOS) 

P 

e. 

Heading angle 

'P 

f. 

Pitch angle 

0 

g- 

Bank angle 

9 

h. 

Control and wing flap position 

5e, 5a, 5r, 5f 

i. 

Body rates and accelerations 

p, q, r, Ax, Ay, Az, 

j- 

Localizer and GS errors 

Localizer and glide slope Theil inequality coefficient 

k. 

State coefficient estimates 

Cma,Cm?,e, C/8a> C„6^ estimate, eiTor, and trut 


Pilot Performance Task Standards 

1. To ensure consistency, flight standards are provided, which will cause pilot gain to be either high or 
low. Low gain will result from the larger error tolerance values, and high gain will result from the 
lower tolerance values. The appropriate standard will be specified on the test card for the maneuver 


event. 

a. Altitude control for level flight: 

+ 150 and + 20 ft 

b. 

Heading control: 

+ 5 ° and + 1 ° 

c. 

Climb/descent rate: 

full power, idle power, power as required per test card 

d. 

Airspeed: 

+ 5 kt and + 1 kt 

e. 

Roll rates: 

+ 10 7sec and + 5 7sec 

f. 

Localizer course intercept angle: 

45 ° and 30 ° 

g- 

Localizer error: 

+ 1 dot and zero lateral deviation -i- 10 kt X wind 

h. 

Glide slope error: 

+ 1 dot and zero vertical deviation -i- 10 kt X wind 


Maneuver Descriptions 

1 . The descriptions of each maneuver below are generalized. Detailed test cards will provide specific 
instructions as to how each maneuver is to be accomplished. Each evaluation will be conducted in 
four levels of turbulence. Icing configuration will be ICE 0 for the initial and final evaluation and ICE 
02 for the instrument approach procedure as shown in Eigure E.l. The 10 kt cross wind in addition to 
the turbulence will be applied for only the instrument approach procedure. Repeat each procedure 3 
times using low gain standards, and then repeat 3 times using high gain standards. Initialization 
conditions will be specified for each maneuver on the test card. 

2. Basic flight maneuvers 

a. Steady level flight at constant heading. 
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(1) Perform at 125 KIAS and 1.3 versus (90 KIAS) with PFLF 

(2) Stabilize on condition with PFLF and call “mark” when stable 

(3) Hold heading and altitude within specified parameters and take data for 3 min 

b. Level turns to headings 

(1) Perform at 125 KIAS and 1.3 versus (90 KIAS) 

(2) Stabilize on condition with PFLF and call “mark” when stable 

(3) Remain steady level for 15 sec. Using power as required maintaining constant airspeed and 
altitude while performing the following: Turn right to a heading of 090 and maintain steady 
level flight for 15 sec. Turn left to 045 and maintain steady flight for 15 sec. Turn left to 030 
and maintain steady level flight for 1 min. 

c. Constant airspeed and vertical speed climbs/descents on a steady heading 

(1) Perform at 125 KIAS and 1.3 versus (90 KIAS) 

(2) Stabilize on condition with PFLF and call “mark” when stable. 

(3) Using power as required, maintain constant airspeed and climb at specified climb rate. Level 
off at 6000 ft and remain steady for 15 sec. Using power as required, begin a constant 
airspeed descent at the specified rate and level off at 5000 ft. Maintain steady level flight for 
15 sec. 

d. Constant airspeed climbing and descending turns to headings 

(1) Perform at 125 KIAS and at 1.3 versus (90 KIAS) 

(2) Remain steady level for 15 sec. Select maximum power and turn right to a heading of 090. 
Maintain constant airspeed in climb and level off at 6000 ft. Stabilize on airspeed and 
heading of 090 for 15 sec. Select idle power and turn left to 360. Maintain constant airspeed 
in the descent and level off at 5000 ft. Stabilize on airspeed and heading for 15 sec. 

(3) Repeat (ii) and use power as required to establish a 300 fpm climb and descent rate 

e. Wing flap transitions 

(1) Initialize on heading 360, PFLF at 85 KIAS, wing flaps up. When stable, call “mark” and 
hold steady condition for 15 sec. Select wing flaps DOWN and use power as required to 
maintain airspeed and altitude while holding heading constant. After wing flaps reach 
DOWN, stabilize and hold the steady condition for 15 sec. 

(2) Stabilize for 1 min with wing flaps full down at 85 KIAS. When stable, call “mark” and 
select wing flaps to UP and use power as required to maintain airspeed while keeping altitude 
and heading constant as the flaps retract to the fully up condition. Stabilize and hold the 
steady condition for 15 sec. 

f. Power transitions: 

(1) Initialize on a 360 heading, at 85 KIAS, wing flaps up, PFLF. When stable, call “mark” and 
rapidly apply full power while maintaining heading and level flight until reaching the 
maximum speed attainable. When stable at maximum speed, hold condition for 15 sec. 

(2) Initialize on a 360 heading, at maximum level flight speed, wing flaps up, PFLF. When 
stable, call “mark” and rapidly reduce power to idle while maintaining heading and altitude. 
When approaching 90 KIAS, add power as required to maintain constant heading and 
airspeed. When stable at 90 KIAS, hold condition for 15 sec. 

g. Operational evaluation — Instrument Approach Procedure 

(1) Initialize with wing flaps up, 3000 ft, heading 360, and 125 KIAS, ICE02 

(2) Conduct the approach procedure per Figure E.l without OBES, and provide pilot alerting 
when state estimates need to be refreshed with additional control activity 

(3) Evaluate workload using NASA TLX 

(4) Repeat the procedure with OBES functioning 

(5) Evaluate workload using NASA TLX 
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Turbulence Levels as Defined by MIL-F-8785C 

1 . Smooth air, no turbulence 

2. Light turbulence 

3. Moderate turbulence 

4. Severe turbulence 

Test Cards 

1 . Detailed test cards will be developed for each of the two sub tasks from the maneuver descriptions 
above. These test cards will be provided in a standard UTSI test template format and retained as part 
of the data package for each test session. Any deviations from initially planned test procedures will be 
noted on the test cards. 

Software Requirements 

1 . D-six executive logic software switch enabling / disabling OBES functionality (BAR) 

2. Alerting the pilot that control excitations are required (BAR) 

a. Update excite button functionality 

b. Apply stop light color scheme 

(1) Tie color scheme to RT-PID confidence bounds 
(a) Parameters in ISP calculation define axis color 
a. Differentiate scheme depending on which axis or axes require excitation 



END 


Figure E.1 . — Test and evaluation profile. Total duration of test is approximately 15 min. Quotes 
are clearances read to the EP, italics are required EP actions per the test protocol. 
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